[skip ci] Fix Iceberg REST S3 path regression coverage - #15422
Merged
Conversation
Signed-off-by: Ray Liu <liurenjie2008@gmail.com>
liurenjie1024
requested review from
gerashegalov and
res-life
and removed request for
a team
July 29, 2026 11:35
Signed-off-by: Ray Liu <liurenjie2008@gmail.com>
Contributor
Greptile SummaryUpdates the Iceberg REST S3 regression test to create a space-containing partition path through Iceberg’s normal writer and verifies that PerfIO S3 was explicitly enabled in the startup Spark configuration.
Confidence Score: 5/5The PR appears safe to merge with no actionable correctness or test-reliability issues identified. The partitioned CTAS matches an existing Iceberg test pattern, the weighted data generator reliably produces the target value, and the REST test launcher explicitly supplies the startup configuration checked by the new assertion. Important Files Changed
Reviews (1): Last reviewed commit: "Revert unnecessary chagnes" | Re-trigger Greptile |
Collaborator
Author
|
We could skip ci since the changed test in this pr will not run in premerge. |
Collaborator
Author
|
build |
8 tasks
nvliyuan
added a commit
that referenced
this pull request
Jul 30, 2026
Resolves the latest merge conflicts in #15412 after #15423 merged. ### Description Merge the current `release/26.08` head into the current `main` head. The initial conflicts were the root and Scala 2.13 project versions. Both were resolved by retaining main's `26.10.0-SNAPSHOT` version instead of the release branch's `26.08.0-SNAPSHOT` version. The branch was refreshed again after additional PRs landed on `release/26.08`. The current diff includes all release updates added after #15423, including: - #15413 — preserve Spark 4.2 BroadcastHashJoin `isSkewJoin` - #15422 — fix Iceberg REST S3 path regression coverage - #15368 — OSS Delta deletion-vector updates - #15411 — fix OSS Delta RTAS on Spark 4.x+ - #15416 — match Spark 4.2 `date_trunc` overflow behavior ### Checklists Documentation - [ ] Updated for new or modified user-facing features or behaviors - [x] No user-facing change Testing - [ ] Added or modified tests to cover new code paths - [x] Covered by existing tests (The included release commits retain their original tests.) - [ ] Not required Performance - [ ] Tests ran and results are added in the PR description - [ ] Issue filed with a link in the PR description - [x] Not required ### Validation - `git diff --check` - Parsed both initially resolved POM files as XML - `python3 -m py_compile` for the modified Iceberg, Delta, and date-time integration tests IMPORTANT: Merge this PR using **Create a merge commit** so the release commit ancestry is preserved and #15412 can close automatically. --------- Signed-off-by: Sameer Raheja <sraheja@.nvidia.com> Signed-off-by: Rahul Prabhu <raprabhu@nvidia.com> Signed-off-by: Chong Gao <chongg@nvidia.com> Signed-off-by: Firestarman <firestarmanllc@gmail.com> Signed-off-by: Ray Liu <liurenjie2008@gmail.com> Signed-off-by: liyuan <yuali@nvidia.com> Signed-off-by: Jihoon Son <ghoonson@gmail.com> Signed-off-by: Niranjan Artal <nartal@nvidia.com> Co-authored-by: Sameer Raheja <sameerz@users.noreply.github.com> Co-authored-by: Sameer Raheja <sraheja@.nvidia.com> Co-authored-by: Gary Shen <gashen@nvidia.com> Co-authored-by: Rahul Prabhu <100436830+sdrp713@users.noreply.github.com> Co-authored-by: Chong Gao <chongg@nvidia.com> Co-authored-by: Chong Gao <res_life@163.com> Co-authored-by: Liangcai Li <firestarmanllc@gmail.com> Co-authored-by: Renjie Liu <liurenjie2008@gmail.com> Co-authored-by: Jihoon Son <ghoonson@gmail.com> Co-authored-by: Niranjan Artal <50492963+nartal1@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #15323.
Description
This is a follow-up to #15360 that makes the Iceberg S3 path regression test reliable with a REST catalog.
The test previously constructed the problematic path by overriding
write.data.pathwith the configured warehouse. For a REST catalog, the warehouse value can be a logical identifier rather than a physical S3 location.The updated test follows
test_iceberg_parquet_read_from_url_encoded_path:spark.rapids.perfio.s3.enabledsetting through SparkConf instead of querying an executor-initialized PerfIO singleton from the driver.Testing performed:
python3 -m py_compile integration_tests/src/main/python/iceberg/iceberg_test.pygit diff --checkA final REST-catalog pipeline rerun is pending.
Checklists
Documentation
Testing
Performance