Skip to content

chore(ci): Advance velox and fix Arrow Flight 18#27163

Merged
czentgr merged 1 commit intoprestodb:masterfrom
czentgr:cz_upgrade_arrow_18
Mar 4, 2026
Merged

chore(ci): Advance velox and fix Arrow Flight 18#27163
czentgr merged 1 commit intoprestodb:masterfrom
czentgr:cz_upgrade_arrow_18

Conversation

@czentgr
Copy link
Copy Markdown
Contributor

@czentgr czentgr commented Feb 18, 2026

  • address arrow flight changes for Arrow 18 to compile with C++20
  • address TempFilePath and TempDirectoryPath refactoring due to Velox PR 16484
  • addresses duplicate signatures with sidecar because key_sampling is now a Velox built-in function

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

  • Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.
  • If adding new dependencies, verified they have an OpenSSF Scorecard score of 5.0 or higher (or obtained explicit TSC approval for lower scores).

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== NO RELEASE NOTE ==

Summary by Sourcery

Update native execution build scripts to apply a custom Arrow Flight patch and adjust Arrow/Flight dependency configuration.

Build:

  • Add support for specifying an EXTRA_ARROW_PATCH for Arrow builds, defaulting to a new arrow-flight.patch file.
  • Bundle absl along with gRPC and Protobuf when building Arrow Flight in native execution.
  • Update Docker dependency image to copy and expose the new Arrow Flight CMake patch via environment variables.

Summary by Sourcery

Update native execution to use a patched Arrow Flight build and align with the latest Velox and SQL-invoked function changes.

Enhancements:

  • Remove the NativeSimpleSamplingPercent SQL-invoked function and its registration from the native SQL-invoked functions plugin.

Build:

  • Allow specifying an EXTRA_ARROW_PATCH for Arrow builds and default it to a new Arrow Flight CMake patch while bundling absl with gRPC and Protobuf for Flight.
  • Copy the Arrow Flight CMake patch into the dependency Docker image and expose it via environment variables for use during setup.

CI:

  • Bump prestodb/presto-native-dependency Docker image tags across native execution, Arrow Flight, and adapter GitHub workflows.

Tests:

  • Adjust native C++ tests to use the new Velox TempFilePath and TempDirectoryPath test utilities and drop the obsolete velox_temp_path dependency.
  • Update native sidecar plugin tests to reflect the removal of the NativeSimpleSamplingPercent function and add coverage for the corresponding join query behavior.

@prestodb-ci prestodb-ci added the from:IBM PR from IBM label Feb 18, 2026
@sourcery-ai
Copy link
Copy Markdown
Contributor

sourcery-ai bot commented Feb 18, 2026

Reviewer's Guide

Updates native execution build and CI configuration to use a patched Arrow Flight 18-compatible dependency image, adjusts Arrow/Flight build options to apply a custom patch and bundle absl, aligns Velox test utilities with upstream refactors, and removes the NativeSimpleSamplingPercent SQL-invoked function and its associated tests/registration.

File-Level Changes

Change Details Files
Make Arrow Flight build use an optional external CMake patch and bundle absl along with other Arrow Flight dependencies.
  • Introduce EXTRA_ARROW_PATCH environment variable with a default pointing to an arrow-flight.patch file when unset
  • Pass absl as a bundled dependency to Arrow’s CMake configuration for Flight builds
  • Use a PRESTO_SCRIPT_DIR variable consistently when sourcing common setup scripts
presto-native-execution/scripts/setup-adapters.sh
Update CI workflows and dependency Docker image to provide the Arrow Flight patch and new Arrow/Velox toolchain.
  • Bump prestodb/presto-native-dependency image tag across native execution, adapters, and Arrow Flight GitHub workflows
  • Copy the arrow-flight.patch file into the dependency Docker image and expose it via the EXTRA_ARROW_PATCH environment variable alongside VELOX_ARROW_CMAKE_PATCH
.github/workflows/prestocpp-linux-build-and-unit-test.yml
.github/workflows/arrow-flight-tests.yml
.github/workflows/prestocpp-linux-adapters-build.yml
presto-native-execution/scripts/dockerfiles/centos-dependency.dockerfile
Align tests and CMake configuration with Velox Temp*Path refactor and remove obsolete test dependency.
  • Switch RemoteFunctionRegisterer tests from exec::test TempFilePath/TempDirectoryPath headers to common::testutil equivalents and update namespaces accordingly
  • Drop the velox_temp_path library from the remote function tests CMake linkage
presto-native-execution/presto_cpp/main/tests/RemoteFunctionRegistererTest.cpp
presto-native-execution/presto_cpp/main/tests/CMakeLists.txt
Remove NativeSimpleSamplingPercent SQL-invoked function and adjust sidecar plugin tests to match the new function set.
  • Stop registering NativeSimpleSamplingPercent in the NativeSqlInvokedFunctionsPlugin
  • Delete the NativeSimpleSamplingPercent implementation class
  • Update sidecar plugin tests: decrement expected inlined function count, add a positive query that should now succeed, and remove an assertion that previously expected a key_sampling_percent registration failure
presto-sql-helpers/presto-native-sql-invoked-functions-plugin/src/main/java/com/facebook/presto/scalar/sql/NativeSqlInvokedFunctionsPlugin.java
presto-native-sidecar-plugin/src/test/java/com/facebook/presto/sidecar/TestNativeSidecarPlugin.java
presto-sql-helpers/presto-native-sql-invoked-functions-plugin/src/main/java/com/facebook/presto/scalar/sql/NativeSimpleSamplingPercent.java
Add Arrow Flight-specific CMake patch file for use during native Arrow builds.
  • Introduce CMake/arrow/arrow-flight.patch to be applied via EXTRA_ARROW_PATCH during Arrow build setup
presto-native-execution/CMake/arrow/arrow-flight.patch

Possibly linked issues

  • #N/A: PR updates remote function tests (TempFile/Directory usage) after Velox changes, aligning with the issue’s failing remote tests

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@czentgr czentgr force-pushed the cz_upgrade_arrow_18 branch from 966ed5d to a7f8a10 Compare February 27, 2026 19:18
@czentgr czentgr changed the title feat(native): Upgrade to use arrow(-flight) 18 chore(ci): Advance velox and fix arrow 18 Feb 27, 2026
@czentgr czentgr force-pushed the cz_upgrade_arrow_18 branch 3 times, most recently from b8658dd to 906a796 Compare February 28, 2026 16:46
@czentgr czentgr changed the title chore(ci): Advance velox and fix arrow 18 chore(ci): Advance velox and fix Arrow Flight 18 Feb 28, 2026
@czentgr czentgr force-pushed the cz_upgrade_arrow_18 branch 2 times, most recently from 612461e to e181b11 Compare March 2, 2026 15:55
@czentgr czentgr marked this pull request as ready for review March 2, 2026 21:21
@czentgr czentgr requested review from a team, pdabre12 and unidevel as code owners March 2, 2026 21:21
@prestodb-ci prestodb-ci requested review from a team and xin-zhang2 and removed request for a team March 2, 2026 21:21
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue, and left some high level feedback:

  • In setup-adapters.sh, consider distinguishing between EXTRA_ARROW_PATCH being unset vs intentionally set to an empty string so that callers can explicitly disable applying any Arrow patch instead of always falling back to arrow-flight.patch when the variable is empty.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In `setup-adapters.sh`, consider distinguishing between `EXTRA_ARROW_PATCH` being unset vs intentionally set to an empty string so that callers can explicitly disable applying any Arrow patch instead of always falling back to `arrow-flight.patch` when the variable is empty.

## Individual Comments

### Comment 1
<location path="presto-native-sidecar-plugin/src/test/java/com/facebook/presto/sidecar/TestNativeSidecarPlugin.java" line_range="569" />
<code_context>
         assertQuery("SELECT any_values_match(MAP(ARRAY[orderkey], ARRAY[totalprice]), k -> abs(k) > 20) from orders");
         assertQuery("SELECT no_values_match(MAP(ARRAY[orderkey], ARRAY[comment]), k -> length(k) > 2) from orders");
         assertQuery("SELECT no_keys_match(MAP(ARRAY[comment], ARRAY[custkey]), k -> ends_with(k, 'a')) from orders");
+        assertQuery("select count(1) FROM lineitem l left JOIN orders o ON l.orderkey = o.orderkey JOIN customer c ON o.custkey = c.custkey");
     }

</code_context>
<issue_to_address>
**suggestion (testing):** Consider also asserting the same query succeeds when the native inlined SQL feature is disabled

Since the failure assertion in `testNonOverriddenInlinedSqlInvokedFunctionsWhenConfigDisabled` was removed, please also add a positive `assertQuery` for this same SQL when the native inlined SQL feature is disabled (either in that test or a new one). This will verify the query now succeeds via the non-native path and ensure coverage for both modes.

Suggested implementation:

```java
        assertQuery("SELECT no_keys_match(MAP(ARRAY[comment], ARRAY[custkey]), k -> ends_with(k, 'a')) from orders");
        assertQuery("select count(1) FROM lineitem l left JOIN orders o ON l.orderkey = o.orderkey JOIN customer c ON o.custkey = c.custkey");
    }

    @Test
    public void testLineitemOrdersCustomerJoinWithInlinedSqlDisabled()
    {
        Session session = createNativeInlinedSqlDisabledSession();
        assertQuery(
                session,
                "select count(1) FROM lineitem l left JOIN orders o ON l.orderkey = o.orderkey JOIN customer c ON o.custkey = c.custkey");
    }

```

To fully wire this up, align the `Session` creation with existing conventions in `TestNativeSidecarPlugin`:

1. If there is already a method like `testNonOverriddenInlinedSqlInvokedFunctionsWhenConfigDisabled`, copy the way it creates a session with native inlined SQL disabled (e.g., a helper method or explicit `Session.builder(getSession())...setSystemProperty(..., "false")...build()`), and replace `createNativeInlinedSqlDisabledSession()` with that pattern.
2. If no such helper exists, implement `createNativeInlinedSqlDisabledSession()` in this class, ensuring it:
   - Starts from the base session used for native execution tests.
   - Sets the system/session property that disables the native inlined SQL feature (use the same property name used elsewhere in this test class for disabling it, or the config property for “inlined sql” in your codebase).
   - Returns the configured `Session`.
3. Ensure the class imports `io.prestosql.Session` (or the correct `Session` class for your Presto fork) if it is not already imported.
4. Optionally, if you prefer to keep all “disabled” assertions in `testNonOverriddenInlinedSqlInvokedFunctionsWhenConfigDisabled`, you can instead:
   - Move the body of `testLineitemOrdersCustomerJoinWithInlinedSqlDisabled` into that existing test method, using its existing `Session` variable, and
   - Remove the new test method introduced above.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@czentgr
Copy link
Copy Markdown
Contributor Author

czentgr commented Mar 2, 2026

For the Iceberg test failures there is a Velox fix available (facebookincubator/velox#16592) that needs to be merged in. Waiting on the update.

@czentgr czentgr force-pushed the cz_upgrade_arrow_18 branch from e181b11 to 0f443b1 Compare March 3, 2026 14:34
@czentgr
Copy link
Copy Markdown
Contributor Author

czentgr commented Mar 3, 2026

@amitkdutta @majetideepak @pdabre12 @tdcmeehan Please review.

@pdabre12
Copy link
Copy Markdown
Contributor

pdabre12 commented Mar 3, 2026

@czentgr I think we run the testOverriddenInlinedSqlInvokedFunctions only with the default behaviour which is inlined_sql_functions set to true.

@sourcery-ai suggests to run it with both inlined_sql_functions enabled and disabled. This comment is okay to ignore for now, we can add that change later.

Copy link
Copy Markdown
Contributor

@pdabre12 pdabre12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @czentgr

Copy link
Copy Markdown
Collaborator

@majetideepak majetideepak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@majetideepak
Copy link
Copy Markdown
Collaborator

@unidevel, @tdcmeehan can one of you please approve this PR? Thanks.

@czentgr czentgr merged commit 86eaf11 into prestodb:master Mar 4, 2026
82 of 83 checks passed
@czentgr czentgr deleted the cz_upgrade_arrow_18 branch March 4, 2026 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

from:IBM PR from IBM

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants