[native] Dynamically Linked Library in Presto CPP by soumiiow · Pull Request #24330 · prestodb/presto

soumiiow · 2025-01-07T17:10:21Z

Description

Depends on facebookincubator/velox#11439 in the Velox space
and based off of the following PR: https://github.com/facebookincubator/velox/pull/1005/files

Motivation and Context

Having these changes will enable users to register custom functions dynamically without requiring a fork of Prestissimo.

Impact

This extends Prestissimo functionality to include dynamic loading of functions, types, connectors, etc.

Test Plan

Unit tested. and Manually end to end tested the changes.

Contributor checklist

Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

General Changes
* ... :pr:`12345`
* ... :pr:`12345`

Hive Connector Changes
* ... :pr:`12345`
* ... :pr:`12345`

If release note is NOT required, use:

== NO RELEASE NOTE ==

linux-foundation-easycla · 2025-01-07T17:10:27Z

The committers listed above are authorized under a signed CLA.

✅ login: soumiiow / name: Soumya Duriseti (24262e3)

czentgr

Please also rebase.

czentgr · 2025-02-03T19:32:42Z

presto-native-execution/presto_cpp/main/CMakeLists.txt

  ${GFLAGS_LIBRARIES}
-  pthread)
+  pthread
+  velox_dynamic_function_loader)


Move this before the velox_encode library.

czentgr · 2025-02-03T19:33:09Z

presto-native-execution/presto_cpp/main/CMakeLists.txt

                                               presto_link_job_pool)

 add_executable(presto_server PrestoMain.cpp)
+target_link_options(presto_server BEFORE PUBLIC "-Wl,-export-dynamic")


Can we add a comment here why we need these flags?

czentgr · 2025-02-03T19:35:24Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

+    const fs::path path(systemConfig->pluginDir());
+    PRESTO_STARTUP_LOG(INFO) << path;
+    std::error_code
+        ec; // For using the non-throwing overloads of functions below.


We don;t need this comment here and so can fix up the odd formatting.

czentgr · 2025-02-03T19:35:52Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

+void PrestoServer::registerDynamicFunctions() {
+  auto systemConfig = SystemConfig::instance();
+  if (!systemConfig->pluginDir().empty()) {
+    // if it is a valid directory, traverse and call dynamic function loader


Please make sure the comments are full sentences beginning with capitalization etc.

czentgr · 2025-02-03T19:37:42Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

+        auto dirEntryPath = dirEntry.path();
+        if (!fs::is_directory(dirEntry, ec) &&
+            extensions.find(dirEntryPath.extension()) != extensions.end()) {
+          facebook::velox::loadDynamicLibrary(dirEntryPath.c_str());


facebook is not needed here because we are already in the facebook namespace.

steveburnett

Thanks for the doc! A few minor formatting and phrasing suggestions.

presto-native-execution/presto_cpp/main/dynamic_registry/README.md

steveburnett

Thanks for the documentation! I really liked your including the setup steps on plugin.rst. Some minor suggestions for formatting and phrasing but looks good overall.

presto-native-execution/presto_cpp/main/dynamic_registry/README.md

presto-docs/src/main/sphinx/presto_cpp/plugin.rst

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

soumiiow · 2025-02-11T22:24:12Z

Thanks @steveburnett please take another look, I've made the changes. I'm not sure about the tone on the intro to UDFs i have on function_plugin.rst, would appreciate another set of eyes there.

steveburnett

Thanks for the revision! Nice work, your unordered list of UDF benefits was great and the format fixes in the README look good.

I made a couple of small suggestions about the intro to UDFs, let me know what you think.

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

steveburnett

I should have noticed this spelling nit earlier! After I found this one, I did a complete review of the doc in this PR and found no other errors so I think this is the last one.

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

aditi-pandit · 2025-06-12T20:19:59Z

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

+* Once defined, easily reusable and called multiple times just like built in functions.
+* Shorter compile times.
+
+1. To create the UDF, create a new C++ file named ``ExampleFunction.cpp`` in the same format as this example:


This reads like you can only use the name ExampleFunction.cpp. You can say something like

"Create a new C++ file for your UDF. We are using a c++ file named `ExampleFunction.cpp`` for the examples below."

good catch! thank you!

To create the UDF, create a new C++ file in the same format as the below example file named ExampleFunction.cpp

hope this conveys the message!! lmk what you think

aditi-pandit · 2025-06-12T20:28:58Z

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

+            }
+      }  
+
+   Note: The ``int64_t`` return type and the ``Varchar`` input type can be changed as needed. Additional or no arguments may be specified as well. For more examples, see the `examples <https://github.com/soumiiow/presto/tree/dylib_new/presto-native-execution/presto_cpp/main/dynamic_registry/examples>`_.


Add a link saying that the functions follow Velox scalar function API. Please follow https://facebookincubator.github.io/velox/develop/scalar-functions.html for more details on how to write scalar functions.

aditi-pandit · 2025-06-12T21:03:18Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

+void PrestoServer::registerDynamicFunctions() {
+  std::error_code
+      ec; // For using the non-throwing overloads of functions below.
+  auto systemConfig = SystemConfig::instance();


aditi-pandit · 2025-06-12T21:05:33Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

+    for (const auto& dirEntry :
+         std::filesystem::directory_iterator(pluginDir)) {
+      if (isSharedLibrary(dirEntry.path())) {
+        PRESTO_STARTUP_LOG(INFO)


Please add the library name in the info message.

czentgr

Looks good!

czentgr · 2025-06-12T22:53:54Z

presto-native-execution/presto_cpp/main/dynamic_registry/examples/DynamicVarcharFunction.cpp

+} // namespace facebook::velox::common::dynamicRegistry
+
+extern "C" {
+


Nit: the other files have no empty line here.

czentgr · 2025-06-12T23:02:23Z

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst

+      };
+
+      extern "C" {
+         void registerExtensions() {


Should we mention it that this name is required in the note below? It is mentioned in the examples.
We could say something like:

The function registerExtensions is the entry point to execute the registration of the UDF and cannot be changed.

i like this, ill add this as a comment for all example files!

majetideepak · 2025-06-13T16:06:42Z

presto-native-execution/presto_cpp/main/dynamic_registry/CMakeLists.txt

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+add_subdirectory(examples)


We need to add a CMake flag PRESTO_ENABLE_EXAMPLES to guard this

will add this here, but i'm wondering should this be able enabled/disabled in presto-native-execution/CMakelists.txt like so:
set(PRESTO_ENABLE_EXAMPLES OFF CACHE BOOL "Enable Presto examples")

presto-native-execution/presto_cpp/main/dynamic_registry/CMakeLists.txt

majetideepak · 2025-06-13T16:08:24Z

presto-native-execution/presto_cpp/main/tests/DynamicFunctionRegistererTest.cpp

@@ -0,0 +1,122 @@
+/*


Is this test being run? I don't see it being added to any test executable.

sorry meant to leave this file out! i mustve picked up this change as i was switching my branches around

majetideepak · 2025-06-13T20:57:48Z

presto-native-execution/CMakeLists.txt

 endif()

+set(PRESTO_ENABLE_EXAMPLES
+    ON


Default OFF

presto-native-execution/presto_cpp/main/dynamic_registry/CMakeLists.txt

majetideepak · 2025-06-16T13:44:07Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp


+void PrestoServer::registerDynamicFunctions() {
+  std::error_code
+      ec; // For using the non-throwing overloads of functions below.


// For using the non-throwing overloads of functions below.
std::error_code ec;

presto-native-execution/presto_cpp/main/CMakeLists.txt

presto-native-execution/presto_cpp/main/PrestoServer.h

aditi-pandit

Thanks @soumiiow for the iterations.

aditi-pandit · 2025-06-18T22:41:45Z

@soumiiow : Can you rebase ?

majetideepak · 2025-06-23T20:19:04Z

@steveburnett can you please re-review the doc changes?

aditi-pandit

Thanks @soumiiow

steveburnett

Looks good! A formatting nit, and a suggestion that I think makes the steps more clear.

presto-docs/src/main/sphinx/presto_cpp/plugin.rst

steveburnett

A single formatting nit, nothing else!

presto-docs/src/main/sphinx/presto_cpp/plugin.rst

steveburnett

LGTM! (docs)

Pull updated branch, new local doc build, looks great. No problems found.

Thanks for your patience!

aditi-pandit

Thanks @soumiiow

steveburnett

LGMT! (docs)

Pull updated branch, new local doc build, everything looks good.

aditi-pandit · 2025-06-25T03:20:58Z

@soumiiow : Can you change the commit title to that of this PR ? Presto PRs are not merged like Velox PRs.

soumiiow self-assigned this Jan 7, 2025

soumiiow force-pushed the dylib_new branch from 58ce37c to aedfb72 Compare January 7, 2025 17:16

soumiiow marked this pull request as ready for review January 7, 2025 18:04

soumiiow requested a review from a team as a code owner January 7, 2025 18:04

tdcmeehan self-assigned this Jan 8, 2025

soumiiow mentioned this pull request Jan 14, 2025

presto requires namespaces when using function registration #24363

Open

tdcmeehan added the from:IBM PR from IBM label Jan 30, 2025

prestodb-ci requested review from a team, pdabre12 and psnv03 and removed request for a team January 30, 2025 18:47

czentgr reviewed Feb 3, 2025

View reviewed changes

soumiiow force-pushed the dylib_new branch from aedfb72 to 9cde5a4 Compare February 3, 2025 21:23

soumiiow requested review from a team and steveburnett as code owners February 3, 2025 21:23

soumiiow requested a review from presto-oss February 3, 2025 21:23

steveburnett requested changes Feb 4, 2025

View reviewed changes

soumiiow force-pushed the dylib_new branch from 9cde5a4 to 689b24e Compare February 11, 2025 09:31

soumiiow requested a review from elharo as a code owner February 11, 2025 09:31

soumiiow requested a review from steveburnett February 11, 2025 09:33

soumiiow force-pushed the dylib_new branch from 689b24e to 6cb96e7 Compare February 11, 2025 10:00

steveburnett requested changes Feb 11, 2025

View reviewed changes

soumiiow force-pushed the dylib_new branch from 6cb96e7 to 0ab37d6 Compare February 11, 2025 22:20

steveburnett requested changes Feb 11, 2025

View reviewed changes

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst Outdated Show resolved Hide resolved

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst Outdated Show resolved Hide resolved

soumiiow force-pushed the dylib_new branch from 0ab37d6 to d8b7dc2 Compare February 12, 2025 18:51

soumiiow requested a review from steveburnett February 12, 2025 18:52

steveburnett requested changes Feb 12, 2025

View reviewed changes

presto-docs/src/main/sphinx/presto_cpp/plugin/function_plugin.rst Outdated Show resolved Hide resolved

soumiiow force-pushed the dylib_new branch from d8b7dc2 to 27bd55e Compare February 12, 2025 19:15

aditi-pandit reviewed Jun 12, 2025

View reviewed changes

czentgr reviewed Jun 12, 2025

View reviewed changes

majetideepak reviewed Jun 13, 2025

View reviewed changes

majetideepak reviewed Jun 16, 2025

View reviewed changes

presto-native-execution/presto_cpp/main/CMakeLists.txt Outdated Show resolved Hide resolved

majetideepak previously approved these changes Jun 16, 2025

View reviewed changes

majetideepak previously approved these changes Jun 18, 2025

View reviewed changes

czentgr reviewed Jun 18, 2025

View reviewed changes

presto-native-execution/presto_cpp/main/PrestoServer.h Show resolved Hide resolved

aditi-pandit previously approved these changes Jun 18, 2025

View reviewed changes

czentgr previously approved these changes Jun 23, 2025

View reviewed changes

majetideepak previously approved these changes Jun 23, 2025

View reviewed changes

aditi-pandit previously approved these changes Jun 23, 2025

View reviewed changes

steveburnett requested changes Jun 23, 2025

View reviewed changes

presto-docs/src/main/sphinx/presto_cpp/plugin.rst Outdated Show resolved Hide resolved

steveburnett previously approved these changes Jun 23, 2025

View reviewed changes

aditi-pandit approved these changes Jun 23, 2025

View reviewed changes

steveburnett approved these changes Jun 24, 2025

View reviewed changes

aditi-pandit mentioned this pull request Jun 25, 2025

Add AI functions #25423

Open

[native] Dynamically Linked Library in Presto CPP

24262e3

This was referenced Aug 19, 2025

Organize Native dynamic functions in catalogs #25832

Open

Write blog article on how to write a dynamic function in Presto Native #25834

Closed

sourcery-ai bot mentioned this pull request Sep 23, 2025

[pos] Refactor catalog config to be inside NativeExecutionConfigModule #26127

Merged

sourcery-ai bot mentioned this pull request Oct 24, 2025

feat(native): Support custom schemas in native sidecar function registry #26236

Merged

6 tasks

Copilot AI mentioned this pull request Nov 4, 2025

Add function catalog support for organizing custom dynamic functions with shared configuration Joe-Abraham/presto#64

Closed

sourcery-ai bot mentioned this pull request Nov 17, 2025

feat(native): Add support for custom execution point in Rest functions #26636

Merged

7 tasks

20001020ycx mentioned this pull request Feb 26, 2026

chore: Integrate upstream Presto release-0.297-edge10 and adapt CLP connector. y-scope/presto#147

Merged

3 tasks

		} // namespace facebook::velox::common::dynamicRegistry

		extern "C" {

Conversation

soumiiow commented Jan 7, 2025

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Release Notes

Uh oh!

linux-foundation-easycla bot commented Jan 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

czentgr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steveburnett left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

steveburnett left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

soumiiow commented Feb 11, 2025

Uh oh!

steveburnett left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

steveburnett left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

soumiiow Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

czentgr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

linux-foundation-easycla bot commented Jan 7, 2025 •

edited

Loading

soumiiow Jun 13, 2025 •

edited

Loading

majetideepak Jun 13, 2025 •

edited

Loading