JSDelivr: Package size exceeded the configured limit of 150 MB.

[KrishnaPG]

What happens?

Running the sample code with CDN is leading to error.

JSDelivr is not serving the package files and instead showing the below error:

Error:

Failed to load the file listing.
Package size exceeded the configured limit of 150 MB.
Consider asking the author to remove unnecessary files to make the package smaller.

To Reproduce

Use the example code with CDN as duckdb.getJsDelivrBundles()

Full code:

  const JSDELIVR_BUNDLES = duckdb.getJsDelivrBundles();

  // Select a bundle based on browser checks
  const bundle = await duckdb.selectBundle(JSDELIVR_BUNDLES);

  const worker_url = URL.createObjectURL(
    new Blob([`importScripts("${bundle.mainWorker}");`], { type: "text/javascript" })
  );

  // Instantiate the asynchronus version of DuckDB-wasm
  const worker = new Worker(worker_url);
  const logger = new duckdb.ConsoleLogger();
  const db = new duckdb.AsyncDuckDB(logger, worker);
  await db.instantiate(bundle.mainModule, bundle.pthreadWorker);
  URL.revokeObjectURL(worker_url);

Browser/Environment:

Edge latest

Device:

PC

DuckDB-Wasm Version:

1.28.1-dev65.0

DuckDB-Wasm Deployment:

jsdelivr

Full Name:

Gopalakrishna Palem

Affiliation:

[carlopi]

Thanks, this was reported on https://discord.duckdb.org, I will be looking for ways to avoid this regression.

[carlopi]

Relevant PR connected to this: #1574, see also the discussion there.

The problem I think here is that there are a lot of functions that are exported (about 80K), and they have on average names that are about 100 characters, and those names are repeated 5 times (1 in the wasm file and 4 in the js file).

80K * 100 * 5 = 40M that is about the overhead we are looking at (multiplied by multiple files since there are [browser | node] [mvp | eh | threads] versions.

@Y--'s solution cuts this to around 10%, that is a great results, but has some brittleness that if possible I would like to avoid.
I am exploring another way that should cut about the same amount, look in the answer to the PR for more details, and long term an improvement of Emscripten is due since most of this overhead is unnecessary.

If there is any feedback, it's welcome.

[mbostock]

To add, this problem is affecting the latest release on npm, e.g.:

https://cdn.jsdelivr.net/npm/@duckdb/duckdb-wasm/+esm

I’m wondering if you intended to make 1.28.1-dev82.0 the current release (@latest tag), or if you intended for this to be a prerelease? Because this version still works:

https://cdn.jsdelivr.net/npm/@duckdb/duckdb-wasm@1.28.0/+esm

If you intend these to be prereleases, I’d recommend publishing them to the next tag, say using npm publish --tag next. (I wrote a guide on this some years back.)

[carlopi]

@mbostock: I like this a lot, I will have next be the tag for development versions. Opened #1591 to track that.

[carlopi]

After #1589 and #1590 package size is now 137.8MB, and jsdelivr is back online: https://www.jsdelivr.com/package/npm/@duckdb/duckdb-wasm

And a cool side effect of the JavaScript side reductions is also that the shell.duckdb.org startup is now faster, even though there are further margin for improvement.

I have also set up a CI job that should compute the size of the package to be uploaded, see for example https://github.com/duckdb/duckdb-wasm/actions/runs/7566857429/job/20605607846#step:22:112, next step is setting up a check that will avoid publishing if the total size is above 150MB / flag this as failure for PRs.