position delete manifest lower_bounds/upper_bounds not correct

[kingwind94]

Query engine

flink 1.12
iceberg 0.13.2

Question

As we know, position delete files should keep the file path and position. In rewriting v2 tables, iceberg would validateNoNewDeletesForDataFiles and would use the new position delete files' lower_bounds/upper_bounds for paths, to check whether the new position deletes should apply to rewritten data files.

However, I found that the lower_bounds/upper_bounds of my position deletes are incompelete as 'hdfs://ns2/user/', which would make iceberg validation method thinks all of the new added position deletes should apply to rewritten data files, and it will fail the rewrite process. Does any one know why the lower_bounds/upper_bounds of position deletes are not correct?

"lower_bounds" : {
  "array" : [ {
    "key" : 2147483546,
    "value" : "hdfs://ns2/user/"
  }, {
    "key" : 2147483545,
    "value" : "\n\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
  } ]
}, 
"upper_bounds" : {
  "array" : [ {
    "key" : 2147483546,
    "value" : "hdfs://ns2/user0"
  }, {
    "key" : 2147483545,
    "value" : "ãå\u0005\u0000\u0000\u0000\u0000\u0000"
  } ]
},

[zhongyujiang]

This is because these metrics were truncated, Iceberg's default metrics mode for column metric is truncate(16). This should be fixed by #6313.
I think it doesn't cause correctness problems, but it does cause more pos delete files to be scanned because the filtering is less effective.

[kingwind94]

This is because these metrics were truncated, Iceberg's default metrics mode for column metric is truncate(16). This should be fixed by #6313. I think it doesn't cause correctness problems, but it does cause more pos delete files to be scanned because the filtering is less effective.

Thx, you are right! I apply #6613 to the flink 1.12 iceberg connector and it also works. The position delete lower_bounds/upper_bounds now keeps full correct path.

Moreover, this problem wont affect correctness issues, but it will fail the rewrite commit validation. I use flink to write data to iceberg and use spark to rewrite small data files and delete files, the problem is every time flink commit a snapshot it will hinder the concurrent rewrite operation beacuse of new flink-added position deletes. But flink's new added position deletes should only appy to the new added data files, not history (rewritting) data files, so this should not hinder the rewrite operation if it works corrctly.
The reason is that DeleteFileIndex.canContainPosDeletesForFile() compares dataFile.path() with posiotion delete's file_path lower_bounds and upper_bounds, which are truncated previous, and this method would always return true for any new position delete, and then hinder the rewrite operation.

[zhongyujiang]

But flink's new added position deletes should only appy to the new added data files, not history (rewritting) data files, so this should not hinder the rewrite operation if it works corrctly.

Unfortunately, 6313 cannot completely solve this problem. It is not enough to use the upper bound and lower bound of file_path to detect whether there is a conflict. Even if metrics are complete, DeleteFileIndex.canContainPosDeletesForFile() still cannot completely filter out unmatched delete files.

There is another PR #5760 trying to fix this.

[kingwind94]

But flink's new added position deletes should only appy to the new added data files, not history (rewritting) data files, so this should not hinder the rewrite operation if it works corrctly.

Unfortunately, 6313 cannot completely solve this problem. It is not enough to use the upper bound and lower bound of file_path to detect whether there is a conflict. Even if metrics are complete, DeleteFileIndex.canContainPosDeletesForFile() still cannot completely filter out unmatched delete files.

There is another PR #5760 trying to fix this.

I think the file_path name of data files are ordered as the flink checkpoint num grows. Not sure about that, but the file_path is not randomly named, at least it seems that DeleteFileIndex.canContainPosDeletesForFile() can filter out new-added position deletes for flink-to-iceberg cases. I will keep looking at it.

And thx again, I will check the PR #5760.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'