Skip to content

test: Add native Iceberg partition transform test cases#26889

Open
PingLiuPing wants to merge 1 commit intoprestodb:masterfrom
PingLiuPing:lp_iceberg_transform_ut
Open

test: Add native Iceberg partition transform test cases#26889
PingLiuPing wants to merge 1 commit intoprestodb:masterfrom
PingLiuPing:lp_iceberg_transform_ut

Conversation

@PingLiuPing
Copy link
Contributor

Description

Add iceberg partition transform testcases.

Motivation and Context

Impact

Test Plan

Contributor checklist

  • Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.
  • If adding new dependencies, verified they have an OpenSSF Scorecard score of 5.0 or higher (or obtained explicit TSC approval for lower scores).

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== NO RELEASE NOTE ==

@prestodb-ci prestodb-ci added the from:IBM PR from IBM label Jan 2, 2026
@sourcery-ai
Copy link
Contributor

sourcery-ai bot commented Jan 2, 2026

Reviewer's Guide

Adds a comprehensive suite of native Iceberg partition transform tests and supporting utilities, including a shared IcebergPartitionTestBase, several focused TestNG test classes for transforms, partitioned writes, schema evolution, and TPCH-based partitioning, plus a new test-scope Hadoop dependency for native execution tests.

Class diagram for new Iceberg native partition test suite

classDiagram
    class IcebergPartitionTestBase

    class TestPartitionTransforms
    class TestPartitionedWrite
    class TestSchemaEvolution
    class TestTpchPartitionTransforms

    IcebergPartitionTestBase <|-- TestPartitionTransforms
    IcebergPartitionTestBase <|-- TestPartitionedWrite
    IcebergPartitionTestBase <|-- TestSchemaEvolution
    IcebergPartitionTestBase <|-- TestTpchPartitionTransforms
Loading

File-Level Changes

Change Details Files
Introduce a reusable Iceberg partition testing base with helpers to create/compare native vs Java Iceberg tables and inspect partition/file metadata.
  • Create IcebergPartitionTestBase extending AbstractTestQueryFramework with native and Java Iceberg query runners
  • Add utilities to build partitioning clauses for identity, bucket, truncate, and temporal transforms
  • Add helpers to create paired native/Java tables, insert test data, compare results, and compute partition metadata via $files and $partitions tables
  • Implement parsing of partition values from file paths and a PartitionInfo record for cross-runner comparison
  • Provide table cleanup helpers for both native and Java runners
presto-native-execution/src/test/java/com/facebook/presto/nativeworker/iceberg/IcebergPartitionTestBase.java
Add targeted tests verifying correctness of partition transforms across data types and multiple partition keys using synthetic data.
  • Create TestPartitionTransforms with shared test_data setup/teardown covering identity, bucket, truncate, and temporal transforms for multiple primitive types
  • Verify that native and Java runners produce identical data, partition values, and partition metadata for each transform
  • Add tests for tables with multiple simultaneous partition transforms and validate partition path structure
presto-native-execution/src/test/java/com/facebook/presto/nativeworker/iceberg/TestPartitionTransforms.java
Add partitioned write tests that exercise various transform combinations on INSERT paths using Iceberg TPCH tables.
  • Create TestPartitionedWrite to populate a source Iceberg table and repeatedly insert into various partitioned target tables
  • Cover identity, bucket (including on DATE), truncate, year/month/day, and combined transforms on different columns
  • Assert row-count preservation and data equality, and validate some partition-specific predicates (e.g., counts per partition value)
presto-native-execution/src/test/java/com/facebook/presto/nativeworker/iceberg/TestPartitionedWrite.java
Add schema evolution tests around adding, renaming, and dropping columns, including partitioned columns and transform-specific partitioning specs.
  • Create TestSchemaEvolution exercising ALTER TABLE ADD COLUMN with identity, truncate, bucket, and temporal partition transforms
  • Test various column renames (including partitioned columns), error cases, and multiple renames while validating data integrity and queryability
  • Add coverage for DECIMAL partitioned columns with different transform types
presto-native-execution/src/test/java/com/facebook/presto/nativeworker/iceberg/TestSchemaEvolution.java
Add TPCH-based partition transform tests that compare native execution against Java for realistic lineitem workloads.
  • Create TestTpchPartitionTransforms that builds Iceberg lineitem tables with different partition transforms (year, bucket on various columns) for both native and Java runners
  • Use helper methods to create and populate paired tables, compare row counts, partition sets, and full data equality
  • Reuse shared partition inspection utilities to validate partition value ranges and partition metadata consistency; keep a multi-transform test stubbed but disabled
presto-native-execution/src/test/java/com/facebook/presto/nativeworker/iceberg/TestTpchPartitionTransforms.java
Wire in a Hadoop Apache test dependency needed by the new native Iceberg tests.
  • Add com.facebook.presto.hadoop:hadoop-apache as a test-scoped dependency in presto-native-execution/pom.xml
presto-native-execution/pom.xml

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@PingLiuPing PingLiuPing force-pushed the lp_iceberg_transform_ut branch from 8adedc8 to 8652bcd Compare January 14, 2026 11:04
@PingLiuPing PingLiuPing marked this pull request as ready for review January 14, 2026 11:04
@PingLiuPing PingLiuPing requested review from a team as code owners January 14, 2026 11:04
@prestodb-ci prestodb-ci requested review from a team, ScrapCodes and nmahadevuni and removed request for a team January 14, 2026 11:04
@PingLiuPing PingLiuPing changed the title test(native): Add Iceberg partition transform test cases. test: Add native Iceberg partition transform test cases. Jan 14, 2026
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • Several helper methods swallow exceptions silently (e.g., dropTableOnJavaRunner and cleanupTables in TestTpchPartitionTransforms); consider at least logging the exception or narrowing the catch to avoid hiding real failures during test setup/teardown.
  • There is some duplicated SQL DDL/insert string construction across tests (e.g., in TestPartitionedWrite and TestTpchPartitionTransforms); extracting common helpers or templates in the base class could make future changes to schemas/partitioning less error‑prone.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- Several helper methods swallow exceptions silently (e.g., dropTableOnJavaRunner and cleanupTables in TestTpchPartitionTransforms); consider at least logging the exception or narrowing the catch to avoid hiding real failures during test setup/teardown.
- There is some duplicated SQL DDL/insert string construction across tests (e.g., in TestPartitionedWrite and TestTpchPartitionTransforms); extracting common helpers or templates in the base class could make future changes to schemas/partitioning less error‑prone.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@PingLiuPing PingLiuPing changed the title test: Add native Iceberg partition transform test cases. test: Add native Iceberg partition transform test cases Jan 15, 2026
@PingLiuPing PingLiuPing force-pushed the lp_iceberg_transform_ut branch from 8652bcd to 55ab559 Compare January 15, 2026 10:14
@PingLiuPing PingLiuPing requested a review from yingsu00 January 22, 2026 11:01
Copy link
Member

@hantangwangd hantangwangd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PingLiuPing thanks for adding these tests. One question: do the tests about varbinary need to wait until facebookincubator/velox#15882 is landed into Prestissimo's Velox before they can pass?

import static com.facebook.presto.testing.assertions.Assert.assertEquals;
import static java.lang.String.format;

public class TestPartitionedWrite
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public class TestPartitionedWrite
@Test(singleThreaded = true)
public class TestPartitionedWrite

nit: the test cases in this class may fail in a concurrent execution environment.


import static java.lang.String.format;

public class TestSchemaEvolution
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public class TestSchemaEvolution
@Test(singleThreaded = true)
public class TestSchemaEvolution

The same as above.

@PingLiuPing
Copy link
Contributor Author

@PingLiuPing thanks for adding these tests. One question: do the tests about varbinary need to wait until facebookincubator/velox#15882 is landed into Prestissimo's Velox before they can pass?

Thank you for the comment, [facebookincubator/velox#15882] has been merged, and I see velox just been advanced today, I will rebase the code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

from:IBM PR from IBM

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants