Tests pass but vitest fails non-deterministically and sometimes causes Unhandled Rejection

[prabhatsharma]

Describe the bug

While test cases results themselves are deterministic but the runs are not. Some of the runs produce unhandled rejection. This seems to be similar to #1191 but not same. My test specs are fairly minimal at this time, like just checking for component rendering and not much.

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

Vitest caught 2 unhandled errors during the test run. This might cause false positive tests.
Please, resolve all the errors to make sure your tests are not affected.

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Rejection ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Error: 
 ❯ post node_modules/vitest/dist/worker.mjs:70:14
 ❯ node_modules/vitest/dist/chunk-vite-node-utils.8077cd3c.mjs:1376:11
 ❯ sendCall node_modules/vitest/dist/chunk-vite-node-utils.8077cd3c.mjs:1373:16
 ❯ node_modules/vitest/dist/chunk-runtime-rpc.9d1f4c48.mjs:7:55
      5|   return new Proxy(rpc2, {
      6|     get(target, p, handler) {
      7|       const sendCall = Reflect.get(target, p, handler);
       |                                                       ^
      8|       const safeSendCall = (...args) => withSafeTimers(() => sendCall(...args));
      9|       safeSendCall.asEvent = sendCall.asEvent;
 ❯ withSafeTimers node_modules/vitest/dist/chunk-utils-global.0a7416cf.mjs:472:20
 ❯ Proxy.safeSendCall node_modules/vitest/dist/chunk-runtime-rpc.9d1f4c48.mjs:7:41
 ❯ process.<anonymous> node_modules/vitest/dist/worker.mjs:34:11
 ❯ process.emit node:events:527:28
 ❯ emit node:internal/process/promises:140:20

Reproduction

You should be able to reproduce it using this repo.

https://github.com/zinclabs/zinc/tree/38303f59af5f329c62068a320eecd4f53c17b107

steps.

git clone https://github.com/zinclabs/zinc
cd zinc
git checkout 38303f59af5f329c62068a320eecd4f53c17b107
cd web
npm i 
npm run test-once

run the test a couple of times. You will notice that sometimes tests are successful however sometimes there is unhandled rejection.

System Info

System:
    OS: macOS 12.4
    CPU: (8) arm64 Apple M2
    Memory: 372.69 MB / 24.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 16.16.0 - /usr/local/bin/node
    npm: 8.11.0 - /usr/local/bin/npm
  Browsers:
    Brave Browser: 103.1.41.96
    Chrome: 103.0.5060.134
    Safari: 15.5
  npmPackages:
    @vitejs/plugin-vue: ^2.3.3 => 2.3.3 
    vite: ^2.9.14 => 2.9.14 
    vitest: ^0.18.1 => 0.18.1

Used Package Manager

npm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[nieyuyao]

This should be caused by some promises not catching errors. For example, the templateService.list() in the getTemplates needs to add a catch.

[rogerleung0411]

May i ask there is any possible solution? same issue here :(

[calvinf]

If you have a vitest setup file via --setupFiles (link), inside of it you can add a very loud failure for unhandledRejection.

// FAIL LOUDLY on unhandled promise rejections / errors
process.on('unhandledRejection', (reason) => {
  // eslint-disable-next-line no-console
  console.log(`FAILED TO HANDLE PROMISE REJECTION`);
  throw reason;
});

This helped me to find the specific file that I was failing to await a Promise that needed to be handled (which was difficult due to hundreds of files/tests).

[sheremet-va]

@calvinf thank you for pointing it out, Vitest didn't serialise errors thrown in unhandledRejection, so some errors might've slipped as Unknown Error: undefined. #2253 should fix it.

[hugw]

@sheremet-va could you tell if the issue fixed relates to my scenario? Facing the same problem right now and I'm using version 0.24.5.
I am randomly getting unhandled rejection errors.
Like 30% of the time the tests pass but get this unhanded rejection that doesn't show exactly which test caused it.

[sheremet-va]

@sheremet-va could you confirm if the issue was fixed? Facing the same problem right now and I'm using version 0.24.5.

I am randomly getting unhandled rejection errors.

Like 30% of the time the tests pass but get this unhanded rejection that doesn't show exactly which test caused it.

Yes, mentioned in this PR error is fixed.

You probably have an asynchronous process that's executed after all tests have passed. Vitest cannot track it, it's unhandled.

[hugw]

I see, thanks for getting back.

[IlyaEremin]

@sheremet-va I'm facing smae issue with unhandled error. What's the strategies to catch it and find a source?

[jtilford]

@sheremet-va I'm facing smae issue with unhandled error. What's the strategies to catch it and find a source?

UPDATE:
I caught additional errors, but this new fix to LogRocket seems to have worked.

Seems to have worked

One of the errors seemed to be related to LogRocket. The error was TypeError: Cannot set properties of undefined (setting '_LRLogger').

So I ended up refactoring my LogRocket import like this:

async function importLogRocket(): Promise<any> {
  if (process.env.VITEST === 'true') return Promise.resolve(null);
  return await import('logrocket');
}

let LogRocket: any = null;

if (process.env.VITEST !== 'true') {
  console.debug('LogRocket init');
  importLogRocket().then((imported_logrocket) => {
    LogRocket = imported_logrocket.default;
    LogRocket.init(LR_ACCOUNT_ID);
  });
}

I'm sure I can refactor further, and probably get rid of the explicit any types for something more specific.

---

Didn't work on its own

I followed the steps above in #1692 (comment)

My vitest has a config file, vitest.config.ts in the root of my project.

The config contains this (truncated example, I removed environment, include, and plugins):

// vitest.config.ts

export default defineConfig({
  test: {
    setupFiles: ['test/vitest/setup-file.ts'],
  }
});

The setup file now has the snippet mentioned above:

// test/vitest/setup-file.ts

// FAIL LOUDLY on unhandled promise rejections / errors
process.on('unhandledRejection', (reason) => {
  // eslint-disable-next-line no-console
  console.log('FAILED TO HANDLE PROMISE REJECTION');
  throw reason;
});

export default {};

(Note the export default {}; at the end, to make it a valid TS file.)

All I can say is, while I don't yet understand why, the non-deterministic failures seem to have disappeared, and I'm also not getting the console log from setup-file.ts.

[kb-0311]

@sheremet-va If I do have some async process running in a test suite and it passes only when all the other tests have passes, else throws this unhandled error as mentioned #1692 (comment) what might be the best way to handle it?

@thoriumdesign did your solution work? did it stop throwing errors non-determinstically ?