Initial work on DuckDB #14114

jdunkerley · 2025-10-06T16:55:26Z

Pull Request Description

Initial set up for DuckDB
Add connection and scaffolding
Using Postgres dialect.

Next PR will add a dialect and bring into unit tests.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

The documentation has been updated, if necessary.
Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
All code follows the
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
Unit tests have been written where possible.
If meaningful changes were made to logic or tests affecting Enso Cloud integration in the libraries,
or the Snowflake database integration, a run of the Extra Tests has been scheduled.
- If applicable, it is suggested to paste a link to a successful run of the Extra Tests.

JaroslavTulach · 2025-10-07T04:14:32Z

why don't we create Standard.Duck instead of putting new 3rd party library into Standard.Database?
At least I believe it has been agreed that we want to move SQLite and Posgress out of Standard.Database eventually
why not use proper architecture since the beginning in the case of DuckDB?
or do we believe DuckDB is so essential it has to be in the core of all databases?

distribution/lib/Standard/Microsoft/0.0.0-dev/docs/api/SQLServer_Connection.md

jdunkerley · 2025-10-07T06:00:07Z

why don't we create Standard.Duck instead of putting new 3rd party library into Standard.Database?

At least I believe it has been agreed that we want to move SQLite and Posgress out of Standard.Database eventually

why not use proper architecture since the beginning in the case of DuckDB?

or do we believe DuckDB is so essential it has to be in the core of all databases?

At this point in testing it, DuckDB is expected to be very core to deliver some of the base functionality, like spatial and some reading and writing of files. If however it doesn't pan out to be useful, I will isolate to a separate library once have gone further along the path.

JaroslavTulach · 2025-10-07T06:46:24Z

deliver some of the base functionality, like spatial

The fact that DuckDB is believed to be essential from product point of view doesn't mean we need to compromise the architecture. It'd be better if none of the core Table&Database concepts (like spatial) were tight to particular implementation.

JaroslavTulach · 2025-10-07T06:55:10Z

why don't we create Standard.Duck instead of putting new 3rd party library into Standard.Database?

Creating new Standard.Duck library from scratch would allow us to more easy validate the Dual NI + JVM Mode for Loading Libraries #13172 approach
We know we can run Test Standard.Microsoft in (mocked) dual JVM mode #14037 in dual JVM mode
but we still haven't turned that mode on in production as Turn Standard.Microsoft into "dual JVM" mode library #13804 requests
- why? because we are afraid of regressing
by creating new Standard.Duck we automatically start executing it in dual JVM mode
that will help us design for dual JVM mode from scratch and help us fine tune everything since beginning
without risking any regressions associated with changing the execution mode later

jdunkerley · 2025-10-07T09:15:50Z

At least how I envision using DuckDB this will be in the NI space for a period of time.

I will put it as its own library and add to the standard list of import - assuming can do quickly as need to move on this.

distribution/lib/Standard/Database/0.0.0-dev/src/Connection/Parquet_Format.enso

Akirathan

I suppose adding DuckDB to NI is OK. I am curious if you will be able to pass the NI size check though.

build.sbt

JaroslavTulach · 2025-10-08T09:53:50Z

build.sbt

            .listFiles("*.jar")
            .map(_.getAbsolutePath()) ++
          `std-tableau-polyglot-root`
+            .listFiles("*.jar")


I would really prefer if we could go without these three lines

it is an excellent opportunity to make dual JVM mode work smoothly

I promise to address any inquiries ASAP

btw. try the experience execute:

sbt:enso> runEngineDistribution --jvm --run test/DuckDB_Tests --vm.D=polyglot.enso.classLoading=Standard.DuckDB:guest,enso_dev.DuckDB_Test:guest,hosted 4 tests succeeded. 0 tests failed. 0 tests skipped. 0 groups skipped.

does it work? Yes, then let's try it. No, tell me what's failing!

Test code:

from Standard.Base import all from Standard.Table import all from Standard.Database import all from Standard.DuckDB import all main = connection5 = Database.connect DuckDB.In_Memory table4 = connection5.execute_query 'SELECT * FROM "/mnt/c/Users/jdunk/Downloads/flights-1m.parquet"' ..All_Rows write_operation=False any10 = table4.aggregate columns=[..Count, ..Maximum 'ARR_DELAY'] any10.display

Data file: https://www.agentsfordata.com/sample-datasets/flights-1m.parquet

distribution/lib/Standard/Database/0.0.0-dev/src/Connection/Connection.enso

JaroslavTulach · 2025-10-08T14:38:40Z

build.sbt

+      "libduckdb_java.so_osx_universal" -> PolyglotLib(MacOSAMD64),
+      "libduckdb_java.so_windows_amd64" -> PolyglotLib(WindowsAMD64),
+      "META-INF/**"                     -> CopyToOutputJar,
+      "org/**/*.class"                  -> CopyToOutputJar


I am getting:

sbt:enso> runEngineDistribution --jvm --run test.enso Execution finished with an error: Can't load library: /enso/build/debug/libduckdb_java.so_linux_amd64 at <java> java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2249) at <java> java.base/java.lang.Runtime.load0(Runtime.java:767) at <java> java.base/java.lang.System.load(System.java:1624) at <java> org.duckdb.DuckDBNative.<clinit>(DuckDBNative.java:52) at <java> org.duckdb.DuckDBConnection.newConnection(DuckDBConnection.java:62) at <java> org.duckdb.DuckDBDriver.connect(DuckDBDriver.java:114) at <java> java.sql/java.sql.DriverManager.getConnection(DriverManager.java:613) at <java> java.sql/java.sql.DriverManager.getConnection(DriverManager.java:160) at <java> org.enso.base.enso_cloud.EnsoSecretHelper.getJDBCConnection(EnsoSecretHelper.java:51)

where the test.enso. We need to patch DuckDBNative.java (upstream as well, please) to use System.loadLibrary...

Yes, CI reports the same (it is masked by ClassNotFoundException)

Related duckdb/duckdb-java#180

JaroslavTulach · 2025-10-08T15:03:25Z

.gitignore

 *.pdb
 *.so
 *.jar
+*.so_linux_amd64


This is wrong idea.

We need .so extension otherwise System.loadLibrary("duckdb") will not find it

but we need to modify DuckDBNative

to call System.loadLibrary first before trying the equilibristic with extracting to File.createTemp

Logic in DuckDBNative appears to have been added to support local dev. I agree that logic is rather flawed if one wants to use the jar as a 3rd party dependency.

JaroslavTulach · 2025-10-09T15:13:18Z

distribution/lib/Standard/DuckDB/0.0.0-dev/src/DuckDB_Connection.enso

+        jdbc_connection = JDBC_Connection.from_java sql_exception=SQLException java_connection
+
+        ## For initial work using DuckDB, we will use the Postgres dialect.
+        DuckDB_Connection.Value (Connection.new jdbc_connection Dialect.postgres DuckDB_Entity_Naming_Properties.new) make_new


using Dialect.postgres doesn't allow us to use the same trick as Standard.Microsoft does

there is SQLServer_Type_Mapping.column_fetcher_factory which loads a class from other JVM

if we did the same for DuckDB, then following worked:

sbt:enso> runEngineDistribution --jvm --run test.enso --vm.D=polyglot.enso.classLoading=Standard.DuckDB:guest,hosted Execution finished with an error: Cannot convert '2006-01-01'(language: Enso Polyglot Bridge, type: java.sql.Date) to Java type 'java.sql.Date': Unsupported target type. at <java> com.oracle.truffle.polyglot.PolyglotObjectProxyHandler.invoke(PolyglotObjectProxyHandler.java:112) at <java> jdk.proxy3/jdk.proxy3.$Proxy46.getDate(Unknown Source) at <java> org.enso.database.JDBCUtils.getLocalDate(JDBCUtils.java:35) at <java> org.enso.database.fetchers.ColumnFetcherFactory$DefaultColumnFetcherFactory$5.getValue(ColumnFetcherFactory.java:72) at <java> org.enso.database.fetchers.BaseColumnFetcher.append(BaseColumnFetcher.java:51) at <java> org.enso.database.fetchers.ColumnFetcher.readResultSet(ColumnFetcher.java:51) at <enso> Result_Set.result_set_to_table.aggregator<arg-2>(Result_Set.enso:38:13-126) at <enso> Result_Set.result_set_to_table.aggregator(Result_Set.enso:37-38) at <enso> Java_Problems.with_problem_aggregator(Java_Problems.enso:116:14-25) at <enso> Result_Set.result_set_to_table(Result_Set.enso:35-39) at <enso> case_branch.statement_setter(Connection.enso:404:25-143)

where test.enso is exactly the snippet being executed

Will be working on a new dialect for DuckDB next

Co-authored-by: Pavel Marek <[email protected]>

DuckDBNative needed to be patched so that it can be loaded as a shared library by name in Native Image. Additionally, it looks like `duckdb_java` didn't follow any standards when it comes to naming the library name: - always prefixed with `lib` (not acceptable on Windows) - suffix includes OS and arch (should be one of `.so`, `.dll` or `.dylib`)

JaroslavTulach · 2025-10-10T15:58:11Z

lib/java/duckdb-wrapper/src/main/java/org/duckdb/DuckDBNative.java

+    static {
+        var libName = "duckdb_java";
+        try {
+            System.loadLibrary(libName);


I'd try to push this System.loadLibrary("duckdb_java") upstream to see how open the project is to 3rd party contributions.

build.sbt

jdunkerley added the CI: No changelog needed Do not require a changelog entry for this PR. label Oct 6, 2025

jdunkerley requested review from 4e6, AdRiley, Akirathan, GregoryTravis, JaroslavTulach and hubertp as code owners October 6, 2025 16:55

github-actions bot added -libs-API-change-Database Marks a PR that changes the public API of Standard.Database -libs-API-change-Microsoft Marks a PR that changes the public API of Standard.Microsoft labels Oct 6, 2025

JaroslavTulach reviewed Oct 7, 2025

View reviewed changes

distribution/lib/Standard/Microsoft/0.0.0-dev/docs/api/SQLServer_Connection.md Show resolved Hide resolved

jdunkerley force-pushed the wip/jd/duckdb-create branch from 31f1fa4 to e8660f8 Compare October 7, 2025 12:24

jdunkerley requested a review from Frizi as a code owner October 7, 2025 12:24

AdRiley approved these changes Oct 7, 2025

View reviewed changes

GregoryTravis approved these changes Oct 7, 2025

View reviewed changes

distribution/lib/Standard/Database/0.0.0-dev/src/Connection/Parquet_Format.enso Outdated Show resolved Hide resolved

JaroslavTulach added the CI: Clean build required CI runners will be cleaned before and after this PR is built. label Oct 7, 2025

github-actions bot added the -libs-API-change-DuckDB Marks a PR that changes the public API of Standard.DuckDB label Oct 7, 2025

Akirathan approved these changes Oct 7, 2025

View reviewed changes

build.sbt Outdated Show resolved Hide resolved

enso-bot bot mentioned this pull request Oct 8, 2025

Removing std-saas libraries from native image #14034

Merged

3 tasks

JaroslavTulach reviewed Oct 8, 2025

View reviewed changes

distribution/lib/Standard/Database/0.0.0-dev/src/Connection/Connection.enso Show resolved Hide resolved

JaroslavTulach reviewed Oct 8, 2025

View reviewed changes

JaroslavTulach reviewed Oct 9, 2025

View reviewed changes

jdunkerley added 2 commits October 9, 2025 16:31

Add DuckDB dependency.

221a142

Initial connectivity to DuckDB.

dedce22

jdunkerley and others added 15 commits October 9, 2025 16:33

lock file

5bef43b

Legal review again

4cd7f70

Docs and empty file.

a2cac72

Bazel lock again

78e4dd0

Fix Bazel issue I hope!

2cb12f9

Add a placeholder Java function for the DuckDB jar.

2cace34

Java format.

2caf3c4

Update build.sbt

8d0fadc

Co-authored-by: Pavel Marek <[email protected]>

Doc update.

f8497b2

Use JDBC within DuckDB library rather than Database.

101d7e0

Fix for setScheam on DuckDB.

db528ec

Extract native libs from DuckDB jar

462ac33

update test: Native lib can have .so.* extension

066ac7f

Ensure duckdb is initialized at runtime

55a364a

Make Internal.DuckDB_Entity_Naming_Properties private

f90b56e

jdunkerley force-pushed the wip/jd/duckdb-create branch from 2452d0e to f90b56e Compare October 9, 2025 15:39

enso-bot bot mentioned this pull request Oct 10, 2025

Run Standard.Google in dual JVM mode #14040

Open

4 tasks

hubertp added 2 commits October 10, 2025 14:54

Add reachability metadata to fix JNI issues

2be3529

JaroslavTulach reviewed Oct 10, 2025

View reviewed changes

build.sbt Show resolved Hide resolved

jdunkerley and others added 2 commits October 12, 2025 15:08

Notices

8f05857

Prettier.

1968721

jdunkerley added the CI: Ready to merge This PR is eligible for automatic merge label Oct 12, 2025

mergify bot merged commit a20924d into develop Oct 12, 2025
81 checks passed

mergify bot deleted the wip/jd/duckdb-create branch October 12, 2025 16:07

hubertp mentioned this pull request Oct 12, 2025

Load DuckDB native library by name first duckdb/duckdb-java#421

Closed

unfurl-links bot mentioned this pull request Oct 13, 2025

Table viz test flaky #14013

Open

Akirathan mentioned this pull request Oct 13, 2025

Investigate Parquet/Avro usage in Enso #8772

Closed

enso-bot bot mentioned this pull request Oct 13, 2025

Improve visualizations feedback via cache observables and safepoints #10525

Open

Initial work on DuckDB #14114

Initial work on DuckDB #14114

Uh oh!

Conversation

jdunkerley commented Oct 6, 2025

Pull Request Description

Checklist

Uh oh!

JaroslavTulach commented Oct 7, 2025

Uh oh!

Uh oh!

jdunkerley commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JaroslavTulach commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JaroslavTulach commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jdunkerley commented Oct 7, 2025

Uh oh!

Uh oh!

Akirathan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JaroslavTulach Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdunkerley Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

jdunkerley Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JaroslavTulach Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

hubertp Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

hubertp Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hubertp Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdunkerley Oct 9, 2025

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jdunkerley commented Oct 7, 2025 •

edited

Loading

JaroslavTulach commented Oct 7, 2025 •

edited

Loading

JaroslavTulach commented Oct 7, 2025 •

edited

Loading

JaroslavTulach Oct 8, 2025 •

edited

Loading

JaroslavTulach Oct 8, 2025 •

edited

Loading

JaroslavTulach Oct 9, 2025 •

edited

Loading

JaroslavTulach Oct 10, 2025 •

edited

Loading