Skip to content
Merged
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.iceberg.actions;

import org.apache.iceberg.expressions.Expression;

/**
* An action for converting the equality delete files to position delete files.
*/
public interface ConvertEqualityDeleteFiles
extends SnapshotUpdate<ConvertEqualityDeleteFiles, ConvertEqualityDeleteFiles.Result> {

/**
* A row filter for finding the equality deletes to convert.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that this should mention that this filter will be converted to a partition filter with an inclusive projection and that matching delete files will be rewritten. It should be clear that it isn't possible to match individual deletes, only whole files.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will add doc.

*
* @param expression An iceberg expression used to find deletes.
* @return this for method chaining
*/
ConvertEqualityDeleteFiles rowFilter(Expression expression);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we have just one filter, there is no need to add "row" in the method name. Let's change it back to filter to match RewriteDataFiles.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


/**
* A partition filter for finding the equality deletes to convert.
*
* @param expression An iceberg expression used to find deletes.
* @return this for method chaining
*/
ConvertEqualityDeleteFiles partitionFilter(Expression expression);

/**
* The action result that contains a summary of the execution.
*/
interface Result {
/**
* Returns the count of the deletes that been converted.
*/
int convertedEqualityDeleteFilesCount();

/**
* Returns the count of the added position delete files.
*/
int addedPositionDeleteFilesCount();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.iceberg.actions;

import org.apache.iceberg.expressions.Expression;

/**
* An action for rewriting the position delete files.
Comment thread
chenjunjiedada marked this conversation as resolved.
Outdated
* <p>
* Generally used for optimizing the sizing and layout of position delete files within a table.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: should be "size" not "sizing"

*/
public interface RewritePositionDeleteFiles
extends SnapshotUpdate<RewritePositionDeleteFiles, RewritePositionDeleteFiles.Result> {

/**
* bin-pack the position deletes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be capitalized.

*
* @return this for method chaining
*/
RewritePositionDeleteFiles binPackPositionDeletes();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why add PositionDeletes to the end of the method name and not just binPack()? Also, if we aren't exposing multiple strategies in the near term, should we simply remove this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense to remove this, ConvertEqualityDeleteFiles already remove the convertEqualityDelete().


/**
* A row filter for finding deletes to rewrite.
*
* @param expression An iceberg expression used to find deletes.
* @return this for method chaining
*/
RewritePositionDeleteFiles rowFilter(Expression expression);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we plan to convert this to a partition filter using an inclusive projection like the equality delete rewrite, correct? @jackye1995, is there a use case for matching files and then using stats to filter out position delete files that don't match? I can't think of one.


/**
* A partition filter for finding deletes to rewrite.
*
* @param expression An iceberg expression used to find deletes.
* @return this for method chaining
*/
RewritePositionDeleteFiles partitionFilter(Expression expression);

/**
* The action result that contains a summary of the execution.
*/
interface Result {
/**
* Returns the count of the position deletes that been rewritten.
*/
int rewrittenDeleteFilesCount();

/**
* Returns the count of the added delete files.
*/
int addedDeleteFilesCount();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.iceberg.actions;

import java.util.Map;
import java.util.Set;
import org.apache.iceberg.DeleteFile;
import org.apache.iceberg.FileScanTask;
import org.apache.iceberg.Table;

/**
* A strategy for the action to convert equality delete to position deletes.
*/
public interface ConvertEqualityDeleteStrategy {

/**
* Returns the name of this convert deletes strategy
*/
String name();

/**
* Returns the table being modified by this convert strategy
*/
Table table();

/**
* Returns a set of options which this convert strategy can use. This is an allowed-list and any options not
* specified here will be rejected at runtime.
*/
Set<String> validOptions();

/**
* Sets options to be used with this strategy
*/
RewriteDeleteStrategy options(Map<String, String> options);

/**
* Select the delete files to convert.
*
* @return iterable of original delete file to be converted.
*/
Iterable<DeleteFile> selectDeleteFiles();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the data file rewrite strategy, the equivalent method accepts an Iterable<FileScanTask> Should this be passed an Iterable<DeleteFile> so that this is selecting from the already loaded files rather than responsible for finding delete files itself?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my POC implementation, there is a local variable that holds the iterable. Accepting the Itrerable<FileScanTask> also sounds reasonable to me. Let up add it.


/**
* Define how to convert the deletes.
*
* @param deleteFilesToConvert a group of files to be converted together
* @return iterable of delete files used to replace the original delete files.
*/
Iterable<DeleteFile> convertDeleteFiles(Iterable<DeleteFile> deleteFilesToConvert);

/**
* Groups delete files into lists which will be processed in a single executable unit. Each group will end up being
* committed as an independent set of changes. This creates the jobs which will eventually be run as by the underlying
* Action.
*
* @param dataFiles iterable of data files that contain the DeleteFile to be converted
* @return iterable of lists of FileScanTasks which will be processed together
*/
Iterable<Iterable<FileScanTask>> planDeleteFileGroups(Iterable<FileScanTask> dataFiles);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: The order of methods doesn't match RewriteStrategy for data files.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will update.

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.iceberg.actions;

import java.util.Map;
import java.util.Set;
import org.apache.iceberg.DeleteFile;
import org.apache.iceberg.Table;

public interface RewriteDeleteStrategy {

@jackye1995 jackye1995 Jul 19, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we also need the methods similar to RewriteStrategy such as name, validOptions

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't add validOptions because we haven't finalized what options are valid and it can be added late. But I'm fine to add them now.

For name, It looks like the class name already includes the strategy name. Do we still need it?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

name is mostly used for logging purpose. I agree it is already in the class name, but for consistency with RewriteStrategy I think we can still have it here.

/**
* Returns the name of this rewrite deletes strategy
*/
String name();

/**
* Returns the table being modified by this rewrite strategy
*/
Table table();

/**
* Returns a set of options which this rewrite strategy can use. This is an allowed-list and any options not
* specified here will be rejected at runtime.
*/
Set<String> validOptions();

/**
* Sets options to be used with this strategy
*/
RewriteDeleteStrategy options(Map<String, String> options);

/**
* Select the delete files to rewrite.
*
* @return iterable of original delete file to be replaced.
*/
Iterable<DeleteFile> selectDeleteFiles();

/**
* Define how to rewrite the deletes.
*
* @param deleteFilesToRewrite a group of files to be rewritten together
* @return iterable of delete files used to replace the original delete files.
*/
Iterable<DeleteFile> rewriteDeleteFiles(Iterable<DeleteFile> deleteFilesToRewrite);

/**
* Groups into lists which will be processed in a single executable unit. Each group will end up being
* committed as an independent set of changes. This creates the jobs which will eventually be run as by the underlying
* Action.
*
* @param deleteFiles iterable of DeleteFile to be rewritten
* @return iterable of lists of FileScanTasks which will be processed together
*/
Iterable<Iterable<DeleteFile>> planDeleteFileGroups(Iterable<DeleteFile> deleteFiles);

@stevenzwu stevenzwu Oct 15, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering why here it is DeleteFile while the ConvertDeleteStrategy plans with FileScanTask.

also wondering if both interfaces in this PR can use RewriteFileGroup as return type? @RussellSpitzer

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is one difference between the rewrite position delete and the convert, it is about the planning.

}