Spark 3.3: Harmonize RewriteDataFilesSparkAction #7676
Conversation
|
Spark-3.2 and older versions have a lot of gaps with spark-3.3 and spark-3.4 (like using FileRewriter instead of RewriteStrategy), hence only backported to spark-3.3 cc: @aokolnychyi, @szehon-ho |
| commitService.start(); | ||
|
|
||
| // Start rewrite tasks | ||
| Collection<FileGroupFailureResult> rewriteFailures = new ConcurrentLinkedQueue<>(); |
There was a problem hiding this comment.
Looks like we are also backporting #7361 to Spark 3.3? No problem with me, but we may want to note that in pr description in case some issues arise because of this.
There was a problem hiding this comment.
you are right. It looks confusing to handle from this PR.
I have reverted it now. It can be done in another PR along with testcase and procedure changes.
ajantha-bhat
changed the title
| }); | ||
|
|
||
| return rewriteFileGroupStream.sorted(rewriteGroupComparator()); | ||
| RewriteExecutionContext ctx, Map<StructLike, List<List<FileScanTask>>> groupsByPartition) { |
There was a problem hiding this comment.
(Not related to this pr). This is my fault, but I think this should be 'StructLikeMap' all across to be safer, I guess we can do it later.
|
Merged, thanks @ajantha-bhat |
2 participants
Simple backport of https://github.com/apache/iceberg/pull/7630/files