test: skip test-diagnostics-channel-memory-leak.js #50327
Conversation
nodejs-github-bot
added
needs-ci
github-actions
bot
removed
the
request-ci
|
Actually, I don't think we can use
|
joyeecheung
changed the title
|
Updated to just remove this test. @jasnell can you take a look again? also cc @Qard @theanarkh |
github-actions
bot
removed
the
request-ci
|
Would checking with FinalizationRegistry help? Could make a ton of instances, register each, and then check if any of them ever triggered the finalizer before the process exits. 🤔 |
That's what I tried with checkIfCollectable() (which does exactly this) and nope it doesn't work, still flaky. Locally the finalizer is triggered after a few instances but in the CI that's a different story. Technically this is also not what FinalizationRegistry guarantees - it's still spec compliant that the finalizer never gets called. I think the lesson learned from #49710 is that FinalizationRegistry with even heap snapshots (which is another inappropriate way to trigger GC) is still unreliable in detecting leaks and leak detection in JS land can only be done as a best-effort. If false positives are inevitable, just don't test it. |
|
I mean, diagnostics_channel itself is using FinalizationRegistry for the cleanup this is supposed to be testing, so if FinalizationRegistry is unreliable isn't that a problem? 🤔 Or is this just an issue of the particular behaviour of --expose_gc? Would there be a way to trigger a deeper GC at the C++ level that we could expose for the tests? |
Yes, which is why FinalizationRegistry should be avoided if possible https://github.com/tc39/proposal-weakrefs#a-note-of-caution - it seems the diagnostics channel implementation is doing something the proposal specifically advise against, though that's out of the scope of this PR.
--expose_gc (or specifically the forced GC it allows) just alters the GC mechanism slightly in that it disables code aging (so compiled code is ever going to be GC'ed, which lead to the failure of the original test once there's more (unrelated) compiled code in the heap). V8 intentionally does not provide a public API to "trigger a deeper GC" (not one that’s guaranteed to clear specific objects that you think should be cleared anyway) because GC is supposed to be internal to the JavaScript engine implementation. I think any test that rely on "a deeper GC" to clear specific objects especially to deflake is not really worth the maintenance cost. If the test doesn't flake, it's okay to keep them, but if they flake, it's better to not do an unreliable flaky test. |
|
If there are no more concerns about removing the test by Wednesday UTC, I am going to land this change to unblock https://chromium-review.googlesource.com/c/v8/v8/+/4962094 - if it's possible to invent a leak detection test that does not flake, it can always be added later, but I think the current test is just flaky in nature, so it's better to remove it for now. |
There was a problem hiding this comment.
Approving to unblock this, but I'd prefer to avoid deleting the test if we can. Would adding the test to a test/**.status file as flaky work rather than deleting it? I don't know enough about how the flaky tagging works to know if that would get things passing while still keeping the test around to unmark in the future if we fix it.
As for diagnostics_channel using FinalizationRegistry, it's not ideal that it might not clean it up, but it was basically a solution to dynamically generated channels (which are discouraged) would otherwise stick around forever. It's an attempt to prevent users from shooting themselves in the foot, but perhaps not 100% successfully. If you've got any ideas on how to do it more reliably I would love to hear them. I think it's probably acceptable as it is though as it's hopefully at least better than definitely leaking. As it is, it just may leak an empty WeakReference instance. 😅
I think in this case having a flaky test is probably not the best solution, because this would be intentionally making the CI more orange. Maybe we can mark it as SKIP directly if we really want to keep the test? |
|
SKIP would be fine too if that gets us to green without throwing out the context that we were testing for this at some point and should probably be finding a better way to test it at some point in the future. 😅 Otherwise, I'm okay with removing the test entirely if there's not a good way to keep it around without it interfering with our CI. Flaky tests are definitely bad tests. 😬 |
There is currently no reliable way to detect this leak because: 1. We cannot reliably get a reference to the channel from the API to detect finalization without creating another strong reference. 2. This test does gc() and then checks memory usage - however the use of gc() disables code aging which can actually lead to increased memory usage overall, as it is not intended to be used to lower memory usage in the first place. 3. The implementation of diagnostics channels relies on ephemeron gc which is inefficient, it's not reliable to use the typical "create a lot of objects and see if it crashes" trick to check leaks. Skip the test for now until we find a way to test it reliably. To avoid flakiness in the CI, it's better to remove an unreliable test altogether.
joyeecheung
changed the title
joyeecheung
added
the
commit-queue
nodejs-github-bot
removed
the
commit-queue
|
Landed in 94156e3 |
|
An interesting idea came up to me about how to test this somewhat more reliably using a (currently somewhat internal) V8 API (v8::debug::QueryObjects) and the CI looks happy with it https://ci.nodejs.org/job/node-test-commit/66307/ - I'll see if it's possible to upstream my V8 patches to expose this to the embedders, some restrictions may need to be added to prevent the embedders from using this too liberally |
|
I'm all for more reliable leak checking. 🙂 |
|
WIP upstream CL in https://chromium-review.googlesource.com/c/v8/v8/+/5006373 |
There is currently no reliable way to detect this leak because: 1. We cannot reliably get a reference to the channel from the API to detect finalization without creating another strong reference. 2. This test does gc() and then checks memory usage - however the use of gc() disables code aging which can actually lead to increased memory usage overall, as it is not intended to be used to lower memory usage in the first place. 3. The implementation of diagnostics channels relies on ephemeron gc which is inefficient, it's not reliable to use the typical "create a lot of objects and see if it crashes" trick to check leaks. Skip the test for now until we find a way to test it reliably. To avoid flakiness in the CI, it's better to remove an unreliable test altogether. PR-URL: nodejs#50327 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>
There is currently no reliable way to detect this leak because: 1. We cannot reliably get a reference to the channel from the API to detect finalization without creating another strong reference. 2. This test does gc() and then checks memory usage - however the use of gc() disables code aging which can actually lead to increased memory usage overall, as it is not intended to be used to lower memory usage in the first place. 3. The implementation of diagnostics channels relies on ephemeron gc which is inefficient, it's not reliable to use the typical "create a lot of objects and see if it crashes" trick to check leaks. Skip the test for now until we find a way to test it reliably. To avoid flakiness in the CI, it's better to remove an unreliable test altogether. PR-URL: #50327 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>
There is currently no reliable way to detect this leak because: 1. We cannot reliably get a reference to the channel from the API to detect finalization without creating another strong reference. 2. This test does gc() and then checks memory usage - however the use of gc() disables code aging which can actually lead to increased memory usage overall, as it is not intended to be used to lower memory usage in the first place. 3. The implementation of diagnostics channels relies on ephemeron gc which is inefficient, it's not reliable to use the typical "create a lot of objects and see if it crashes" trick to check leaks. Skip the test for now until we find a way to test it reliably. To avoid flakiness in the CI, it's better to remove an unreliable test altogether. PR-URL: #50327 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>
There is currently no reliable way to detect this leak because: 1. We cannot reliably get a reference to the channel from the API to detect finalization without creating another strong reference. 2. This test does gc() and then checks memory usage - however the use of gc() disables code aging which can actually lead to increased memory usage overall, as it is not intended to be used to lower memory usage in the first place. 3. The implementation of diagnostics channels relies on ephemeron gc which is inefficient, it's not reliable to use the typical "create a lot of objects and see if it crashes" trick to check leaks. Skip the test for now until we find a way to test it reliably. To avoid flakiness in the CI, it's better to remove an unreliable test altogether. PR-URL: #50327 Reviewed-By: James M Snell <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>
* chore: bump node in DEPS to v20.11.0 * module: bootstrap module loaders in shadow realm nodejs/node#48655 * src: add commit hash shorthand in zlib version nodejs/node#50158 * v8,tools: expose necessary V8 defines nodejs/node#50820 * esm: do not call getSource when format is commonjs nodejs/node#50465 * esm: fallback to readFileSync when source is nullish nodejs/node#50825 * vm: allow dynamic import with a referrer realm nodejs/node#50360 * test: skip test-diagnostics-channel-memory-leak.js nodejs/node#50327 * esm: do not call getSource when format is commonjs nodejs/node#50465 * lib: fix assert throwing different error messages in ESM and CJS nodejs/node#50634 * src: fix compatility with upcoming V8 12.1 APIs nodejs/node#50709 * deps: update base64 to 0.5.1 nodejs/node#50629 * src: avoid silent coercion to signed/unsigned int nodejs/node#50663 * src: fix compatility with upcoming V8 12.1 APIs nodejs/node#50709 * chore: fix patch indices * chore: update patches * test: disable TLS cipher test This can't be enabled owing to BoringSSL incompatibilities. nodejs/node#50186 * fix: check for Buffer and global definition in shadow realm nodejs/node#51239 * test: disable parallel/test-shadow-realm-custom-loader Incompatible with our asar logic, resulting in the following failure: > Failed to CompileAndCall electron script: electron/js2c/asar_bundle * chore: remove deleted parallel/test-crypto-modp1-error test * test: make test-node-output-v8-warning generic nodejs/node#50421 * chore: fixup ModuleWrap patch * test: match wpt/streams/transferable/transform-stream-members.any.js to upstream * fix: sandbox is not enabled on arm * chore: disable v8 sandbox on ia32/arm --------- Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com> Co-authored-by: Shelley Vohr <[email protected]> Co-authored-by: Cheng Zhao <[email protected]>
test: skip test-diagnostics-channel-memory-leak.js
There is currently no reliable way to detect this leak because:
API to detect finalization without creating another strong reference.
use of gc() disables code aging which can actually lead to increased
memory usage overall, as it is not intended to be used to lower
memory usage in the first place.
which is inefficient, it's not reliable to use the typical "create
a lot of objects and see if it crashes" trick to check leaks.
Skip the test for now until we find a way to test it reliably.
To avoid flakiness in the CI, it's better to remove an unreliable
test altogether.
I found this issue from an integration test run in https://chromium-review.googlesource.com/c/v8/v8/+/4962094