Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate persistent Windows test failure on test.wasi/test-return-on-exit #37374

Closed
Trott opened this issue Feb 15, 2021 · 12 comments
Closed
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI. v8 engine Issues and PRs related to the V8 dependency. wasi Issues and PRs related to the WebAssembly System Interface. windows Issues and PRs related to the Windows platform.

Comments

@Trott
Copy link
Member

Trott commented Feb 15, 2021

  • Test: test.wasi/test-return-on-exit
  • Platform: win2012r2
  • Console Output:
00:30:50 not ok 778 wasi/test-return-on-exit # TODO : Fix flaky test
00:30:50   ---
00:30:50   duration_ms: 49.248
00:30:50   severity: flaky
00:30:50   exitcode: 134
00:30:50   stack: |-
00:30:50     (node:972) ExperimentalWarning: WASI is an experimental feature. This feature could change at any time
00:30:50     (Use `node --trace-warnings ...` to show where the warning was created)
00:30:50     
00:30:50     <--- Last few GCs --->
00:30:50     
00:30:50     
00:30:50     <--- JS stacktrace --->
00:30:50     
00:30:50     FATAL ERROR: Zone Allocation failed - process out of memory
00:30:50   ...
  • Build Links:

https://ci.nodejs.org/job/node-test-binary-windows-js-suites/8229/RUN_SUBSET=3,nodes=win2012r2-COMPILED_BY-vs2019-x86/console and pretty much any other recent Jenkins CI run as the failure is persistent.

@Trott
Copy link
Member Author

Trott commented Feb 15, 2021

Refs: #36139 (comment)

@Trott
Copy link
Member Author

Trott commented Feb 15, 2021

@nodejs/platform-windows

@Trott Trott added flaky-test Issues and PRs related to the tests with unstable failures on the CI. wasi Issues and PRs related to the WebAssembly System Interface. windows Issues and PRs related to the Windows platform. labels Feb 15, 2021
@Trott
Copy link
Member Author

Trott commented Feb 15, 2021

Since this apparently started happening with the V8 update to 8.8: @nodejs/v8

@Trott Trott added the v8 engine Issues and PRs related to the V8 dependency. label Feb 15, 2021
@targos
Copy link
Member

targos commented Feb 15, 2021

Also @nodejs/wasi

@targos
Copy link
Member

targos commented Feb 22, 2021

There's probably a real bug somewhere, because this only happens on a 32 bit system.

@Trott
Copy link
Member Author

Trott commented Feb 28, 2021

There's probably a real bug somewhere, because this only happens on a 32 bit system.

Any thoughts on how we can make some progress on this?

@Trott
Copy link
Member Author

Trott commented Feb 28, 2021

Refs: #36139 (comment)

@gengjiawen Any ideas on what to do here?

@Trott
Copy link
Member Author

Trott commented Feb 28, 2021

@nodejs/testing Any ideas for what we might be able to do in CI or elsewhere to figure this out?

@cjihrig
Copy link
Contributor

cjihrig commented Feb 28, 2021

I agree that this is probably a genuine bug since it only shows up on 32-bit Windows, and coincided with the V8 8.8 update (which had wasm related changes). The returnOnExit feature also relies on somewhat corner case behavior - monkey patching the WASI import to throw a JavaScript exception that WebAssembly can't catch.

Regarding the test - as a last resort, we could skip it on Windows. It looks like there are two test cases in test/wasi/test-return-on-exit.js. Can we identify which of them is causing the crash, or if splitting them into separate test files somehow mitigates the problem?

@targos
Copy link
Member

targos commented Mar 1, 2021

https://ci.nodejs.org/job/node-test-commit-windows-fanned/41469/

Windows CI with targos@55e5b94

@cjihrig
Copy link
Contributor

cjihrig commented Mar 1, 2021

I re-ran the Windows CI a couple times as well. All failures were in test-return-on-exit-1, which contains the following code:

// Flags: --experimental-wasi-unstable-preview1
'use strict';
const common = require('../common');
const assert = require('assert');
const fs = require('fs');
const path = require('path');
const { WASI } = require('wasi');
const wasmDir = path.join(__dirname, 'wasm');
const modulePath = path.join(wasmDir, 'exitcode.wasm');
const buffer = fs.readFileSync(modulePath);

(async () => {
  // Verify that if a WASI application throws an exception, Node rethrows it
  // properly.
  const wasi = new WASI({ returnOnExit: true });
  wasi.wasiImport.proc_exit = () => { throw new Error('test error'); };
  const importObject = { wasi_snapshot_preview1: wasi.wasiImport };
  const { instance } = await WebAssembly.instantiate(buffer, importObject);

  assert.throws(() => {
    wasi.start(instance);
  }, /^Error: test error$/);
})().then(common.mustCall());

The main difference here is that wasi.wasiImport.proc_exit() is monkey-patched a second time here to throw an Error object, rather than throwing a Symbol as returnOnExit would normally.

@cjihrig
Copy link
Contributor

cjihrig commented Mar 1, 2021

I noticed another small difference that shouldn't have any impact. In the flaky test, the proc_exit() function is not bound to the WASI import. I ran cjihrig@32cb5b7 through the CI a few times, and haven't seen this test fail. Maybe I'm doing something wrong? If that does fix the flakiness, this seems like a probable bug in V8.

cjihrig added a commit to cjihrig/node that referenced this issue Mar 5, 2021
Starting with the V8 8.8 update, this test has been regularly
crashing with an out of memory error on 32-bit Windows. The issue
has been narrowed down to a function not being bound. This seems
like a V8 bug, but at least it seems that we can work around it.

Fixes: nodejs#37374
@Trott Trott closed this as completed in 8ab6a6c Mar 8, 2021
danielleadams pushed a commit that referenced this issue Mar 16, 2021
Starting with the V8 8.8 update, this test has been regularly
crashing with an out of memory error on 32-bit Windows. The issue
has been narrowed down to a function not being bound. This seems
like a V8 bug, but at least it seems that we can work around it.

Fixes: #37374

PR-URL: #37615
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Michaël Zasso <[email protected]>
targos pushed a commit that referenced this issue May 1, 2021
Starting with the V8 8.8 update, this test has been regularly
crashing with an out of memory error on 32-bit Windows. The issue
has been narrowed down to a function not being bound. This seems
like a V8 bug, but at least it seems that we can work around it.

Fixes: #37374

PR-URL: #37615
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Michaël Zasso <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI. v8 engine Issues and PRs related to the V8 dependency. wasi Issues and PRs related to the WebAssembly System Interface. windows Issues and PRs related to the Windows platform.
Projects
None yet
3 participants