Benchmarks against Node.js Test Runner and Bun's test runner

[BoscoDomingo]

Hello, I just came across this nice project but at my company we use either Node's or Bun's built-in test runners, so I was wondering what the pros are for us to move to Poku (if it even makes sense in the first place).

I think adding benchmarks against said runners would give a better understanding of what makes this a better product

[wellwelwel]

Hi @BoscoDomingo, first of all, thanks 🙋🏻‍♂️

I'll answer in categories:

Performance

About built-in test runners, I don't believe that other test runners are comparable when it comes to performance exclusively. For example, they don't necessarily need to compile the native features to start running the tests.

---

Compatibility

As a library maintainer, in my case it's important to ensure compatibility with previous Node.js versions. The native Node.js test runner ceased to be experimental since v20.

Poku (2.x.x) supports Node.js from version 8 onwards (although this isn't really a goal).

---

Cross-platform

Poku also allows you to ensure that the same project is compatible with Node.js, Bun and Deno through the same test suite for both.

In your case where you use different test runners, even if they are not in the same project, the teams could easily create tests in the same "syntax".

---

JavaScript "natural" Syntax

Unlike most test runners, instead of something like:

describe('My Test', { concurrency: 1 }, () => {
  // You need to explicitly state what should be run before the tests
  beforeAll(() => {
    console.log('Started');
  });

  // The last step of the script is called before the tests themselves
  afterAll(() => {
    console.log('Done');
  });

  // Asynchronous tests, but they will be executed sequentially even without the use of `await`
  it(async () => {
    // async test
  });

  it(async () => {
    // async test
  });
});

Poku allows you to use your knowledge of JavaScript in tests:

import { describe, it } from 'poku';

describe('My Test', async () => {
  console.log('Started');

  await it(async () => {
    // async test
  });

  await it(async () => {
    // async test
  });

  console.log('Done');
});

• In addition to the syntax philosophy, Poku also offers hooks such as beforeEach and afterEach.

---

I think these are important considerations when making decisions in this case.

Note

These points can be found in more detail at https://poku.io/docs/philosophy.

[wellwelwel]

It's also worth mentioning the limitations, especially the absence of Stubs, Mocks and Spies, including no TAP reporters compatibility and no built-in coverage report 🙋🏻‍♂️

There is a plan to include some of these features, but with no specific deadline.

[BoscoDomingo]

I see your point, and it's totally valid. Maybe adding a "Why not the built-in solutions" with exactly what you included here could be useful for people like me then ;)

I would suggest benchmarking against Node and Bun in any case. If they aren't too far off, then you can say "look, you can get cross-compatibility without losing much performance in the process" which would be a major selling point to anyone who needs it or prefers writing tests in the way you allow.

And if they are, then you're are at least informing potential users of shortcomings before they commit to your product, which is extremely welcome and in line with what you're already doing.

Just my two cents, though!

[wellwelwel]

Naturally, Node.js is expected to be faster, but I didn't expect to see such a small difference:

Important

Note that the benchmark comparison is quite simple/basic.

Jest           x 2.39 ops/sec — ±0.62% (11 runs sampled)
Mocha + Chai   x 5.53 ops/sec — ±1.59% (18 runs sampled)
Vitest         x 2.42 ops/sec — ±1.11% (11 runs sampled)
Poku           x 10.29 ops/sec — ±0.62% (30 runs sampled)
Node           x 10.76 ops/sec — ±1.33% (31 runs sampled)

🚀 Fastest is Node

For Bun, there is a bigger difference:

Running Poku through Bun.

Poku (Bun)     x 16.57 ops/sec — ±1.87% (45 runs sampled)
Bun            x 41.62 ops/sec — ±1.92% (56 runs sampled)

🚀 Fastest is Bun

About Deno, it can't run CommonJS to perform a real/fair benchmark, but I think the difference could be a middle ground between Node.js and Bun results, considering its performance:

Deno           x 24.98 ops/sec — ±0.49% (46 runs sampled)

---

Note

As the benchmarks are run every CI and always expects Poku to "win", I don't intend to include the native benchmarks in the continuous benchmarks, but it was interesting to compare results (especially when it comes to Node.js).

I'll use this issue itself to link this question in the documentation as a FAQ.

Thanks, @BoscoDomingo 🙋🏻‍♂️

[BoscoDomingo]

Nice! Those are some great results, especially compared to the Node Test Runner

[mrazauskas]

It is interesting to ask which Node.js version was used to benchmark the native test runner? And if warmup runs were performed?

[wellwelwel]

Sorry, I forgot to mention these details.

• Node.js: v22.8.0
• Bun: 1.1.26
• Deno: 1.45.5

About warmup, the benchmark approach is exactly the same from benchmark, but using:

• Node.js: node:test + node:assert
• Bun: bun:test + expect
• Deno: Deno.test + assertEquals

I want to migrate the benchmarks to hyperfine, following your recommendation in #584 (comment).

---

This section of Node.js documentation may explain the results. Apparently, the approach is practically the same as that used in Poku (I was quite surprised).