My (so far unsucessful) attempt to integrate duckdb-wasm with esbuild

[llimllib]

Hi! I have an app that duckdb-wasm would be perfect for, I saw your blog post, and I'm failing to figure out how to use it. I've kept a detailed log of what I've done here, and if anybody would be so kind as to help me figure out how to get it installed I'd love to contribute some good getting started documentation back to the project.

The app is my NBA statistics viewer; right now it loads json files into js objects and allows users to eval javascript statements against a javascript object as a poor man's query, but it would be much nicer to have a database such as duckdb-wasm.

Currently, the app is just a single javascript file run directly by the browser, so I figured I'd also need to start compiling the file; I wanted to use esbuild because who in their right mind would choose to use webpack, and searching this repository seemed to suggest esbuild was usable with duckdb-wasm.

I started by just adding an import to my javascript file:

import * as duckdb from '@duckdb/duckdb-wasm/dist/duckdb-esm.js';

Then I npm installed it and esbuild:

npm i @duckdb/duckdb-wasm esbuild

And added a wasm target to my makefile for compiling the file:

dist/viewer_duckdb.js: duckdb.js
	./node_modules/.bin/esbuild viewer.js --bundle --outfile=dist/viewer_duckdb.js

wasm: dist/viewer_duckdb.js

Unfortunately, this failed with what looked like a dependency error:

./node_modules/.bin/esbuild duckdb.js --bundle --outfile=out.js
 > node_modules/@duckdb/duckdb-wasm/dist/duckdb-esm.js:1:75: error: Could not resolve "apache-arrow" (mark it as external to exclude it from the bundle)
    1 │ ...mWriter as ee,Table as U}from"apache-arrow";import{AsyncByteQueue as Z}f...
      ╵                                 ~~~~~~~~~~~~~~

1 error
make: *** [out.js] Error 1

So I tried installing all the dependencies:

npm i apache-arrow crypto os fs path fast-glob wasm-feature-detect stream

Now the file builds! 🎉

However, when I import the resulting file into my HTML file, it throws an error:

Uncaught TypeError: class heritage stream_1.Readable is not an object or null
    js http://devd.io:8001/dist/viewer_duckdb.js:13475
    __require http://devd.io:8001/dist/viewer_duckdb.js:10
    <anonymous> http://devd.io:8001/dist/viewer_duckdb.js:15606
    <anonymous> http://devd.io:8001/dist/viewer_duckdb.js:16582
viewer_duckdb.js:13475:30

I don't have a good guess as to what to do here, so I'm pretty stuck. Does anybody have suggestions about what I'm doing wrong?

I've published the branch where I am trying to get this to build - make wasm will build the file if you want to try it out yourself

edit 1:

this build script works for compiling my script.

I have a working database but I am having a heck of a time figuring out how to actually load data into or use the result objects that come back out of the database

[domoritz]

There is an issue with the 6.0 release of arrow that I fixed in master and I'm waiting for a release. You can try the esm version or wait for the fix to be released. Hopefully that fixes things for your app.

[ankoh]

Thanks for trying it out, Bill!

duckdb-wasm depends on apache-arrow 6 that has a broken default bundle at the moment.
It will import a node io writer and introduce dependencies to, for example, "stream" that won't work in the browser.
We're facing the very same problem with the duckdb-wasm-shell.

The bundle is already fixed in the arrow master. (thanks @domoritz )
So you could either wait for arrow 6.0.1, which should be pushed out very soon OR you install

        "@apache-arrow/esnext-esm": "^6.0.0",

instead of apache-arrow as dependency.
https://github.com/duckdb/duckdb-wasm/blob/master/packages/duckdb-wasm-shell/package.json#L23

Let us know how this is going!

[llimllib]

Thanks! installing that returned me to the first error I was seeing:

 > node_modules/@duckdb/duckdb-wasm/dist/duckdb-esm.js:1:75: error: Could not resolve "apache-arrow" (mark it as external to exclude it from the bundle)
    1 │ import{RecordBatchReader as F,RecordBatchStreamWriter as ee,Table as U}from"apache-arrow";import{AsyncByteQueue as Z}from"apache-arrow";var P=class extends Z{flush(){let e=super.toUint8Array(!0);return this._values.length=0,e}};var N=class{constructor(e,r){this._bindings=e,this._conn=r}close(){this._bindings.disconnect(this._conn)}useUnsafe(e...
      ╵                                                                            ~~~~~~~~~~~~~~

It's not quite clear to me why esbuild can't import that version of apache-arrow as apache-arrow, but most javascript packaging is pretty confusing to me so I'll wait for the 6.0.1 to get released I guess

[llimllib]

Just to link the issue in question, it appears to be this issue resolved by this commit

[llimllib]

Ok I learned more than I ever desired to about building stuff with esbuild and now I can build the package with this build script

[ankoh]

Haha :). Very awesome! Yes overriding the resolving is also the way with webpack but far from ideal. I hope we can update to 6.0.1 soon.

[ankoh]

Btw. make sure to link your project in "Show and tell" when it is ready!

[llimllib]

will do!

[bmschmidt]

On a similar topic, so tagging along with a build-related problem. I think my question boils down to--do you have a recommended way to install the latest arrow version manually? Like a commit number or something?

For my bundler setup (svelte-kit package, so vite for dev and rollup for production) I couldn't get it to work using "@apache-arrow/esnext-esm": "^6.0.0" ("No matching export in "browser-external:stream" for import "Duplex"") so I just built the latest version of arrow-js from apache/arrow main branch. (package.json calls it ("version": "7.0.0-SNAPSHOT")) and dumped it into my node_modules by hand.

That almost works--in some cases I can run queries--but I run into various weird errors. (In dev, 64-bit ints are showing up as an array of two 32-bit ints in the arrow tables; building with svelte's adapter-static, some problems where it's calling node modules that can't exist).

[domoritz]

I got annoyed by this as well and switched to 32 bit ints to floats in my app. In Arrow 7 we can hopefully switch entirely to bigints.

[ankoh]

Hm all these issues with 6.0.0 are really annoying, I'm sorry. I hope we can update to 6.0.1 quickly.
I will include minimal examples for the most popular bundlers once 6.0.1 lands, they really should be part of the CI.

With webpack, the following works:

resolve: {
    alias: {
        'apache-arrow': path.resolve(__dirname, './node_modules/apache-arrow/Arrow.esnext.min.js'),
    },
},

Which is the same as the override in the esbuild script of @llimllib.

[ankoh]

If you don't want BigInt64Arrays even though your browser supports it, you can disable it through
https://shell.duckdb.org/docs/interfaces/index.DuckDBConfig.html#emitBigInt
via https://shell.duckdb.org/docs/classes/index.AsyncDuckDB.html#open

We cast BIGINT result columns as doubles in that case.

[domoritz]

Here is the Arrow 6.0.1 pull request: apache/arrow#11628

[bmschmidt]

Perfect, thanks @ankoh and @domoritz. (And of course for your work on the library as well!)

This with the newest version seems to be enough for me to get it working in svelte-kit with adapter-static in both dev in prod. (Not one of the most common bundlers, obviously).

my svelte.config.js contents:

/** @type {import('@sveltejs/kit').Config} */
import path from 'path'
import { fileURLToPath } from 'url';
import { dirname } from 'path';
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);

import adapter from '@sveltejs/adapter-static'

const config = {
	kit: {
		// hydrate the <div id="svelte"> element in src/app.html
		target: '#svelte',
		adapter: adapter()
	},
	vite: {
		resolve: {
			"alias": {
				"apache-arrow" : path.resolve(__dirname, './node_modules/apache-arrow/Arrow.esnext.min.js'),
			}
		}
	}
}

export default config;