Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(cmd-api-server): fix flaky runtime-plugin-imports test #1667

Closed
petermetz opened this issue Dec 17, 2021 · 0 comments · Fixed by #1903
Closed

test(cmd-api-server): fix flaky runtime-plugin-imports test #1667

petermetz opened this issue Dec 17, 2021 · 0 comments · Fixed by #1903
Assignees
Labels
API_Server bug Something isn't working dependencies Pull requests that update a dependency file Flaky-Test-Automation Issues related to test stability (which is a long running issue that can never fully be solved) good-first-issue Good for newcomers good-first-issue-400-expert Tests Anything related to tests be that automatic or manual, integration or unit, etc.

Comments

@petermetz
Copy link
Contributor

petermetz commented Dec 17, 2021

This is a flaky test case meaning that it cannot be reliably reproduced, it only happens
every now and then which of course makes it harder to fix as well.

There is a clue I found based on the error message printed by Jest though:
https://stackoverflow.com/a/50793993

Logs

logs_12177.zip

ReferenceError: You are trying to `import` a file after the Jest environment has been torn down. From packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts.

      346 |
      347 |       // eslint-disable-next-line @typescript-eslint/no-var-requires
    > 348 |       const pluginPackage = require(/* webpackIgnore: true */ packagePath);
          |                             ^
      349 |       const createPluginFactory = pluginPackage.createPluginFactory as PluginFactoryFactory;
      350 |       const pluginFactoryOptions: IPluginFactoryOptions = {
      351 |         pluginImportType: pluginImport.type,

      at ApiServer.instantiatePlugin (packages/cactus-cmd-api-server/src/main/typescript/api-server.ts:348:29)
          at runMicrotasks (<anonymous>)
      at ApiServer.initPluginRegistry (packages/cactus-cmd-api-server/src/main/typescript/api-server.ts:313:22)
      at ApiServer.getOrInitPluginRegistry (packages/cactus-cmd-api-server/src/main/typescript/api-server.ts:289:33)
@petermetz petermetz added API_Server bug Something isn't working dependencies Pull requests that update a dependency file Flaky-Test-Automation Issues related to test stability (which is a long running issue that can never fully be solved) good-first-issue Good for newcomers good-first-issue-400-expert Tests Anything related to tests be that automatic or manual, integration or unit, etc. labels Dec 17, 2021
@Leeyoungone Leeyoungone self-assigned this Mar 8, 2022
Leeyoungone added a commit to Leeyoungone/cactus that referenced this issue Mar 8, 2022
Leeyoungone added a commit to Leeyoungone/cactus that referenced this issue Mar 25, 2022
Leeyoungone added a commit to Leeyoungone/cactus that referenced this issue Mar 25, 2022
Leeyoungone added a commit to Leeyoungone/cactus that referenced this issue Mar 25, 2022
Leeyoungone pushed a commit to Leeyoungone/cactus that referenced this issue Mar 25, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

This is a PARTIAL resolution to issue hyperledger-cacti#238

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>

Fixes hyperledger-cacti#1667
Leeyoungone pushed a commit to Leeyoungone/cactus that referenced this issue Mar 29, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

This is a PARTIAL resolution to issue hyperledger-cacti#238

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>

Fixes hyperledger-cacti#1667
Leeyoungone pushed a commit to Leeyoungone/cactus that referenced this issue Mar 30, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

This is a PARTIAL resolution to issue hyperledger-cacti#238

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>

Fixes hyperledger-cacti#1667
Leeyoungone pushed a commit to Leeyoungone/cactus that referenced this issue Apr 5, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

This is a PARTIAL resolution to issue hyperledger-cacti#238

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>

Fixes hyperledger-cacti#1667
Leeyoungone pushed a commit to Leeyoungone/cactus that referenced this issue Apr 5, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

This is a PARTIAL resolution to issue hyperledger-cacti#238

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>

Fixes hyperledger-cacti#1667
Leeyoungone pushed a commit to Leeyoungone/cactus that referenced this issue Apr 6, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

This is a PARTIAL resolution to issue hyperledger-cacti#238

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>

Fixes hyperledger-cacti#1667
Leeyoungone pushed a commit to Leeyoungone/cactus that referenced this issue Apr 20, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

This is a PARTIAL resolution to issue hyperledger-cacti#238

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>

Fixes hyperledger-cacti#1667
Leeyoungone pushed a commit to Leeyoungone/cactus that referenced this issue Apr 25, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

This is a PARTIAL resolution to issue hyperledger-cacti#238

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>

Fixes hyperledger-cacti#1667
Leeyoungone pushed a commit to Leeyoungone/cactus that referenced this issue May 9, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

This is a PARTIAL resolution to issue hyperledger-cacti#238

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>

Fixes hyperledger-cacti#1667
Leeyoungone pushed a commit to Leeyoungone/cactus that referenced this issue May 9, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

This is a PARTIAL resolution to issue hyperledger-cacti#238

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>

Fixes hyperledger-cacti#1667
Leeyoungone pushed a commit to Leeyoungone/cactus that referenced this issue May 9, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

This is a PARTIAL resolution to issue hyperledger-cacti#238

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>

Fixes hyperledger-cacti#1667
Leeyoungone pushed a commit to Leeyoungone/cactus that referenced this issue May 9, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

This is a PARTIAL resolution to issue hyperledger-cacti#238

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>

Fixes hyperledger-cacti#1667
petermetz pushed a commit to Leeyoungone/cactus that referenced this issue May 9, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

=======================================================================

This is a flaky test case that can't be reliably reproduce. I've ran the
test individually 40 times and all 40 times it has passed successfully.

This error, however, seems to pop up only when the `yarn jest` command
is run (when I ran the test individually).

Based on the linked
["hint"](https://stackoverflow.com/a/50793993)
from the issue, it seems like using `jest.useFakeTimers()` a good
solution. However, using the `jest.useFakeTimers()` with`async/await`
was not recommended.

However, I found a [different source]
(https://gist.github.com/apieceofbart/e6dea8d884d29cf88cdb54ef14ddbcc4#file-test-js-L58)
that showed examples
of how to use the `jest.useFakeTimers()`.
Our original problem, however, is probably because the after the test
environment is torn down, the second test file is trying to import
files from the first environment, triggering the error
([explained here](jestjs/jest#11438 (comment))).

So I've come to the solution of installing another node package:
[node-cleanup](https://www.npmjs.com/package/node-cleanup) and running
that within the afterAll at the very end of the test.

My laptop can't take running the `yarn jest` script too much so I
haven't seen it reproduce the error yet. (WOT ruhrow)

_______________________edit below _________________________

So it turns out that this flaky test due to libraries trying to third
libraries trying to run after the test runs is a simple fix that was
documented [here](https://testing-library.com/docs/using-fake-timers/)
where we just add the `useRealTimer` in the `afterEach`. Lol so sorry!!

=======================================================================

This is a PARTIAL resolution to issue hyperledger-cacti#238

Fixes hyperledger-cacti#1667

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>
Signed-off-by: Peter Somogyvari <[email protected]>
petermetz pushed a commit that referenced this issue May 10, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

=======================================================================

This is a flaky test case that can't be reliably reproduce. I've ran the
test individually 40 times and all 40 times it has passed successfully.

This error, however, seems to pop up only when the `yarn jest` command
is run (when I ran the test individually).

Based on the linked
["hint"](https://stackoverflow.com/a/50793993)
from the issue, it seems like using `jest.useFakeTimers()` a good
solution. However, using the `jest.useFakeTimers()` with`async/await`
was not recommended.

However, I found a [different source]
(https://gist.github.com/apieceofbart/e6dea8d884d29cf88cdb54ef14ddbcc4#file-test-js-L58)
that showed examples
of how to use the `jest.useFakeTimers()`.
Our original problem, however, is probably because the after the test
environment is torn down, the second test file is trying to import
files from the first environment, triggering the error
([explained here](jestjs/jest#11438 (comment))).

So I've come to the solution of installing another node package:
[node-cleanup](https://www.npmjs.com/package/node-cleanup) and running
that within the afterAll at the very end of the test.

My laptop can't take running the `yarn jest` script too much so I
haven't seen it reproduce the error yet. (WOT ruhrow)

_______________________edit below _________________________

So it turns out that this flaky test due to libraries trying to third
libraries trying to run after the test runs is a simple fix that was
documented [here](https://testing-library.com/docs/using-fake-timers/)
where we just add the `useRealTimer` in the `afterEach`. Lol so sorry!!

=======================================================================

This is a PARTIAL resolution to issue #238

Fixes #1667

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>
Signed-off-by: Peter Somogyvari <[email protected]>
zondervancalvez pushed a commit to zondervancalvez/cactus that referenced this issue May 18, 2022
Migrated test from Tap to Jest.

File Path:
packages/cactus-test-cmd-api-server/src/test/typescript/integration/runtime-plugin-imports.test.ts

=======================================================================

This is a flaky test case that can't be reliably reproduce. I've ran the
test individually 40 times and all 40 times it has passed successfully.

This error, however, seems to pop up only when the `yarn jest` command
is run (when I ran the test individually).

Based on the linked
["hint"](https://stackoverflow.com/a/50793993)
from the issue, it seems like using `jest.useFakeTimers()` a good
solution. However, using the `jest.useFakeTimers()` with`async/await`
was not recommended.

However, I found a [different source]
(https://gist.github.com/apieceofbart/e6dea8d884d29cf88cdb54ef14ddbcc4#file-test-js-L58)
that showed examples
of how to use the `jest.useFakeTimers()`.
Our original problem, however, is probably because the after the test
environment is torn down, the second test file is trying to import
files from the first environment, triggering the error
([explained here](jestjs/jest#11438 (comment))).

So I've come to the solution of installing another node package:
[node-cleanup](https://www.npmjs.com/package/node-cleanup) and running
that within the afterAll at the very end of the test.

My laptop can't take running the `yarn jest` script too much so I
haven't seen it reproduce the error yet. (WOT ruhrow)

_______________________edit below _________________________

So it turns out that this flaky test due to libraries trying to third
libraries trying to run after the test runs is a simple fix that was
documented [here](https://testing-library.com/docs/using-fake-timers/)
where we just add the `useRealTimer` in the `afterEach`. Lol so sorry!!

=======================================================================

This is a PARTIAL resolution to issue hyperledger-cacti#238

Fixes hyperledger-cacti#1667

Signed-off-by: awadhana <[email protected]>
Signed-off-by: Youngone Lee <[email protected]>
Signed-off-by: Peter Somogyvari <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API_Server bug Something isn't working dependencies Pull requests that update a dependency file Flaky-Test-Automation Issues related to test stability (which is a long running issue that can never fully be solved) good-first-issue Good for newcomers good-first-issue-400-expert Tests Anything related to tests be that automatic or manual, integration or unit, etc.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants