'eCommerce Promotion Tracking' TSVB vis causes Possible EventTarget memory leak detected.

[LeeDr]

Kibana version: 8.3.0 (main source)

Elasticsearch version: 8.3.0

Server OS version: OSX

Browser version: Edge (Chromium)

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.):
node scripts/functional_tests_server.js --config x-pack/test/functional/config.base.js

Describe the bug: A TSVB visualizations included with the eCommerce sample data set logs a warning and stack trace about "Possible EventTarget memory leak detected."

Steps to reproduce:

1. Install eCommerce sample data set
2. watch the log while you open the [eCommerce] Promotion Tracking visualization.
3. The visualization looks fine. When I look at Last 7 days in timepicker I see 4 lines and annotations.

Expected behavior: Opening the visualization shouldn't cause warnings with stack traces.

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

proc [kibana] (node:5768) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 11 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 proc [kibana] [2022-05-19T16:39:01.580-05:00][WARN ][process] MaxListenersExceededWarning: Possible EventTarget memory leak detected. 11 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
 proc [kibana]     at AbortSignal.[kNewListener] (node:internal/event_target:430:17)
 proc [kibana]     at AbortSignal.[kNewListener] (node:internal/abort_controller:172:24)
 proc [kibana]     at AbortSignal.addEventListener (node:internal/event_target:539:23)
 proc [kibana]     at /Users/leedr/git/main/kibana/node_modules/@elastic/transport/src/connection/HttpConnection.ts:122:24
 proc [kibana]     at new Promise (<anonymous>)
 proc [kibana]     at HttpConnection.request (/Users/leedr/git/main/kibana/node_modules/@elastic/transport/src/connection/HttpConnection.ts:97:18)
 proc [kibana]     at KibanaTransport.request (/Users/leedr/git/main/kibana/node_modules/@elastic/transport/src/Transport.ts:462:67)
 proc [kibana]     at AsyncHooksRunContextManager.with (/Users/leedr/git/main/kibana/node_modules/elastic-apm-node/lib/instrumentation/run-context/BasicRunContextManager.js:49:17)
 proc [kibana]     at Instrumentation.withRunContext (/Users/leedr/git/main/kibana/node_modules/elastic-apm-node/lib/instrumentation/index.js:579:30)
 proc [kibana]     at KibanaTransport.request (/Users/leedr/git/main/kibana/node_modules/elastic-apm-node/lib/instrumentation/modules/@elastic/elasticsearch.js:231:33)
 proc [kibana]     at KibanaTransport.request (/Users/leedr/git/main/kibana/src/core/server/elasticsearch/client/create_transport.ts:57:29)
 proc [kibana]     at AsyncSearch.submit (/Users/leedr/git/main/kibana/node_modules/@elastic/elasticsearch/src/api/api/async_search.ts:156:33)
 proc [kibana]     at search (/Users/leedr/git/main/kibana/src/plugins/data/server/search/strategies/ese_search/ese_search_strategy.ts:75:36)
 proc [kibana]     at runMicrotasks (<anonymous>)
 proc [kibana]     at processTicksAndRejections (node:internal/process/task_queues:96:5)

Any additional context:
It seems that if I remove just one of the series from the visualization the warning stops.

[elasticmachine]

Pinging @elastic/kibana-vis-editors @elastic/kibana-vis-editors-external (Team:VisEditors)

[flash1293]

I think it's related to abort listeners and the fact that TSVB is doing a bunch of searches in parallel. See also #125197 (comment)

@pgayvallet @delvedor Is there something we should/could do different here? I don't think it's an actual issue but if even a single user can trigger this limit then it's likely it will occur a lot on a busy Kibana server (and 10 seems like a low limit for a global pool of abort listeners). Should we bump it to 100 or so?

[pgayvallet]

I'm not sure. I would have expected elastic/elastic-transport-js#42 to fix the problem upstream tbh. @delvedor do you think we could be doing anything wrong here, or as @flash1293 said, it's just normal to reach the default limit under heavy load and we should just increase it?

[flash1293]

if there is one listener per request then dozens of them at the same time are definitely expected. Relabelling to core team as it seems like this isn't related to the TSVB integration - if that turns out to be wrong please relabel.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[delvedor]

Hello! If you are reusing the abort signal multiple times, the client is cleaning up listeners as soon as they are no longer needed, but if you are sending more than 10 concurrent requests with the same abort signal, then you will hit that "issue" again.
I'm using quotes around issue because that's just a warning. Node.js has no way to know if having dozens of listeners is ok, so it's proactive and lets you know because sometimes it can mean a memory leak.

If it's expected to share the same abort signal among multiple concurrent request, you can change the max listeners limit with:

• You can increase the max listeners globally with https://nodejs.org/api/events.html#eventsdefaultmaxlisteners
• While you can increase the max listeners for a single abort signal with https://nodejs.org/api/events.html#eventssetmaxlistenersn-eventtargets

[flash1293]

Thanks, that helps. So I tested around a bit and it seems like the search service attaches 3 listeners per search. As a single TSVB request can do multiple searches, it's going quickly above that limit. Adding

    setMaxListeners(esRequests.length*3, abortSignal);

to the TSVB code silences the warning, but it's a little annoying it needs to know about implementation details of the search service to do so (why 3?). I'm a little on the fence on how to fix this. To get rid of the warning I'm going to add the above line to the service for now but maybe we can improve.