Reduce Build Times

[epreston]

Reduce build times by replacing a number of rollup plugins with rollup/esbuild.

Description

Build / build-watch times can be reduced by a 10x. Currently, builds take 9 seconds for some products on high end machines and run sequentially. The could be in the high 800 ms and run concurrently.

This does not address the development web server, serve, which locks files preventing builds from completing, requiring complex logic to stop and start processes. That can be resolved by switching to a dev server designed for live editing / dev. This is just raw built time in isolation.

The rollup/esbuild plugin can process both typescript and javascript (or a mixture of the two). It can produce type declarations during this process. Leveraging this may simplify the current set of build commands, complete some requests from devs. In short, simplify tooling and remove a few chores.

Since it creates and processes the AST for source maps, its easy to get meaningful references for debugging in the same pass. This means no juggling sourcemaps for typescript.

Steps to Reproduce

1. build the library
2. attempt to fight off the intrusive thoughts for 30 seconds

Resolution Proposed

• PR to introduce rollup/esbuild plugin and update build commands to produce the same outputs.
• Remove unused plugins, consolidate build commands
• Before and after analysis on build times and build products
• If paired with a replacement to "serve", this can provide the live coding experience with the benefits of build products.

[mvaligursky]

Relevant discussions:
#2103
#3575

[epreston]

I am currently running this proposed setup for my own projects; similar to how vue, snowpack, and amazon leverage it.

It's popular among well known projects in the wider javascript ecosystem. It's well established over the last 3 years. Currently, this is the best in class and is the underlying technology for simplicity and performance from higher level build tools (rollup / webpack / vite).

It works by moving the heavy string processing required for transformation / reference / ast creation / polyfil identification and insertion / away from babel and rollup into highly optimised code. That AST is shared with rollup so all the normal rollup build logic applies. In short, the build system is rollup with a turbo charger.

It eats ts and js.

[marklundin]

Hi @epreston, thanks for raising this, it's a great suggestion. When you say rollup/esbuild, do you see a combination of the two working together similar to Vite? Would be great to understand your thoughts on this.

[epreston]

This does not remove all previous tools or limit the ones that can be used in the rollup ecosystem.

The playcanvas projects don't have these requirements at the moment but;

• Some libraries are very particular about the formatting of typescript enums translated to javascript. They have a special build command for this which leverages slower tools prior to publishing to the NPM registry.
• Some libraries still use tsc with api-extractor prior to publishing to the NPM registry to trim and simply type definition even further.

Terser can be used directly so the 1% to 3% build size benefit from using slower tools has evaporated.

[epreston]

@marklundin yes kinda, but in the way that vue/core uses rollup with rollup/esbuild plugin. They leverage vite for instant live coding. I publish projects on this account that show this in action. I feel weird linking them because my personal goal is to promote playcanvas.

I will have the PR completed in a few hours. I've drafted it many times. I've been slowly improving the engine build system so the PR clean and easy to review.

We could replace all the build tooling using rollups recommended "managed configuration", aka vite. That would simplify the entire build system to 10 lines for the same outputs, provide a web server, documentation, a community to support the build config, and would work consistently over all public playcanvas projects.

It has one major issue: I would struggle to find something to contribute. I suggested it at the start so no guilt. 😄

While that all may be true, there's still 20% more we can squeeze from tools. I want to fight for that 20% and push the boundaries at least on the "engine" project.

[epreston]

@marklundin sometimes code just says it better. Here's the basics.

Simplified rollup.config.js example psudo code:

import esbuild from 'rollup-plugin-esbuild';

const output = {
  sourcemap = true,
  minify = true,
  minifyWhitespace = true,
  target = 'es2022'
}

function resolveDefines() {
   // replace jscc for defines
} 

export default {
  input: 'src/index.js',
  plugins: [
    //... other plugins, preprocessing
    esbuild({
        tsconfig: path.resolve(__dirname, 'tsconfig.json'),
        sourceMap: output.sourcemap,
        minify: output.minify,
        minifyWhitespace: output.minifyWhitespace,
        target: output.target,
        define: resolveDefines()
    }),
    //... other plugins, terser, etc
  ],
  //...other options
}

See: https://www.npmjs.com/package/rollup-plugin-esbuild

You slide it in, wire things up so everything is the same externally, add a few new options. Advanced example next.

[epreston]

If you want to build every published playcanvas project at the same time in under 2 seconds, you do the following. Want it faster, add a build cache. Still want it faster, hire me.

This is a build.js pseudo code script, adapted from vue/core, that that makes the rollup -c command async.

import { execa, execaSync } from 'execa';
import { cpus } from 'node:os';

run();

async function run() {
  try {
    const resolvedTargets = targets.length
      ? fuzzyMatchTarget(targets, buildAllMatching)
      : allTargets;
    await buildAll(resolvedTargets);
  } finally {
    // removeCache();
  }
}

async function buildAll(targets) {
  // await runParallel(cpus().length, targets, build);  // <-- uncomment me
  await runParallel(1, targets, build);
}

async function runParallel(maxConcurrency, source, iteratorFn) {
  const ret = [];
  const executing = [];
  for (const item of source) {
    const p = Promise.resolve().then(() => iteratorFn(item, source));
    ret.push(p);

    if (maxConcurrency <= source.length) {
      const e = p.then(() => executing.splice(executing.indexOf(e), 1));
      executing.push(e);
      if (executing.length >= maxConcurrency) {
        await Promise.race(executing);
      }
    }
  }
  return Promise.all(ret);
}

async function build(target) {

  // if building a specific format, do not remove dist.
  if (!formats && existsSync(`${pkgDir}/dist`)) {
    await fs.rm(`${pkgDir}/dist`, { recursive: true });
  }

  const env =
    (pkg.buildOptions && pkg.buildOptions.env) || (devOnly ? 'development' : 'production');

  await execa(
    'rollup',
    [
      // '--experimentalLogSideEffects',
      '-c',
      '--environment',
      [
        `TARGET:${target}`,
        sourceMap ? `SOURCE_MAP:true` : ``
      ]
        .filter(Boolean)
        .join(',')
    ],
    { stdio: 'inherit' }
  );
}

Instead of having rollup -c in the package.json, we point to a utils/build.js script. This will spawn a process according to the number of CPUs. With large fans and watercooling in the PC, it's helped me keep warm during these cold winter months.

Console output suffers.

My public projects use a working version of this.

[marklundin]

Thanks for this @epreston, really fantastic information. I was literally just looking at the 'rollup-plugin-esbuild' as I hadn't seen it before.

One thing I noticed is that esbuild doesn't target ES5 and I know we have an old ES5 target. Maybe @willeastcott can chime in on that

[epreston]

@marklundin the current "es5" build target is well beyond the ES5 language standard. its a common point of confusion. When people say ES5 in this project, its more shorthand for "without modules, in a single file, wrapped in a named global".

I wrote up a something along this with analysis. Building for the es5 language standard is not sensible nor does it have an real world intersection with physical devices.

#5697

[Maksims]

Reducing build times - is an amazing task to tackle. The way web-dev ended up, with all these preprocessing, transpiling, building, etc - is ridiculous.

I believe there is no strict official lowest platform/device/feature-set to be supported, but it basically tries to support "every possible device" that supports WebGL. Obviously at this time it might be not sensible to support iPhone 5S, but still. New features were slowly whitelisted based on their adoption by browser vendors.
And I believe it is a good approach. At the same time you can adjust the engine build the way it suits your needs, which is great.

[epreston]

@Maksims it can be small and simple with 90% of the benefits. This is overkill for nearly every project except popular libraries.

Truth be told, the PR should simplify the current build process.

Strip out the nice to haves / boiler plate from the following project and you have 3 lines in the package.json and 7 lines in the build config. All singing and dancing, automatic, with no developer burden, all modern features. Something you can just dump spaghetti code and unoptimized resources into and get quality outcomes.

https://github.com/epreston/template-web-playcanvas

We can actually do numbers and analysis to know what support needs to be. You're right about the slow automatic whitelisting. Every tool in this project currently supports that style of build config. Please read the analysis linked above.

I'll give you a teaser of one of the fun insights: continuing to support webgl1 only increases browser support by 1% in Uganda in practical terms. (when you take the full intersection of devices in use capable of running the engine). Wonder why its being dropped from other libraries with webgl2 being universal and webgpu taking the spotlight.

Fighting for that last bit of perf is a mark of quality. This is why I love web-based real time 3d. Choices matter. Each improvement means more sprites on screen, 5 more minutes of playtime, more complex shaders.

Thank you for reading. I'm excited to share some ideas. Maybe some of them stick.

[epreston]

First pass: 9 seconds down to 2.5 seconds. Target is 0.37 seconds. more info
See PR draft for details.

[LeXXik]

I would like to note that the main goal of reducing the build time is not for reducing the time per se, but for the developer experience. As such, anything that is not perceptable by a human dev (like target 0.37) should be of less importance. I would personally be happy with anything sub second.