Rewrite plugin_status logic limiting usage of Observables (reducing heap size) by gsoldevila · Pull Request #128324 · elastic/kibana

gsoldevila · 2022-03-22T22:27:02Z

Attempt to solve #128061

The idea is to rewrite the src/core/server/status/plugins_status.ts logic in order to simplify the RxJS pipeline. We take into account "input" observables and "output" observables, but we no longer use RxJS for the internal update of plugin statuses.

We receive core status updates, and plugin reported status updates through input observables. We debounce these events to avoid unnecessary calculations.
The PluginsStatusService class maintains an updated copy of the state, in a sort of Redux store pattern, so to speak. The state includes convenient structures to make updates more efficient. These structures include a tree of reverse dependencies, and a list of plugin names sorted by depth. The root plugins (depth = 1) are those that do not depend on any other plugin. A plugin that depends on a root plugin will have depth = 2, and so on so forth.
The state is updated iteratively, going through the ordered list of plugins. This causes less recursive calls and less heap consumption.
Once the state has been updated, PluginsStatusService then signals the internal Subjects that are connected to the output observables.

Key changes to reduce heap size and improve performance

Previous implementation used Observables to recalculate and propagate the status of each plugin through the tree of dependencies. The switchMap and concatMap operators create new Observables with each higher order emission.
The recursive approach is not optimal in this scenario, because in order to make sure all statuses are updated, we might need to visit each node more than once. For instance, if plugin C depends on plugins A and B we might have to calculate C's status twice (once when we process A's dependencies and another time when we process B's dependencies).

Memory consumption impact

Heap size after Kibana started reduced from 800Mb to 280Mb.
Heap snapshot is taken right after Kibana started.
main branch

the current branch.

mshustov · 2022-03-23T16:15:54Z

buildkite build this

mshustov · 2022-03-23T16:27:26Z

buildkite test this

mshustov · 2022-03-23T17:00:42Z

@gsoldevila, if we confirm the positive impact of the PR on memory consumption, I'd consider porting it back to 8.1 and 7.x.

jbudz

This is huge, memory consumption looks great:

#128061 (comment) for comparison.

mshustov · 2022-03-24T10:56:09Z

+      this.coreStatus = coreStatus!;
+      // console.log('⚡ CORE STATUS! elastic: ', coreStatus.elasticsearch.level.toString(), '; savedObjects: ', coreStatus.savedObjects.level.toString());
+      const derivedStatus = getSummaryStatus(Object.entries(this.coreStatus), {
+        allAvailableSummary: `All dependencies are available`,


Maybe we shouldn't pass the default values?

This was like that in the previous implementation.
The getSummaryStatus is used in a couple of places:

status_service uses the default "All services are available"

plugins_status (this class) uses the slightly different "All dependencies are available".

The problem is that getSummaryStatus method does not currently have the knowledge of whether it's being called with plugins statuses only, or with plugins statuses + core services statuses.

But if we agree to use the exact same message for both cases, then I can probably remove that default value.

mshustov · 2022-03-24T11:40:46Z

+
+  private updatePluginsStatuses(plugins: PluginName[]): void {
+    const toCheck = new Set<PluginName>(plugins);
+    for (let i = 0; i < this.orderedPluginNames.length && toCheck.size > 0; ++i) {


question: maybe we should iterate through toCheck set?

Actually, we need to iterate in "depth" order:

first updating root plugins' statuses, then updating depth=2, and so on so forth.

Otherwise we might try to update a plugin status before updating its dependencies.

elasticmachine · 2022-03-24T18:43:30Z

Pinging @elastic/kibana-core (Team:Core)

kibana-ci · 2022-03-25T10:19:29Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💛 Build #33645 was flaky 3d64fed
💔 Build #33463 failed c1f15f3
💔 Build #33126 failed 6113711
💔 Build #32952 failed 6113711
💔 Build #32859 failed 4b023f4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…eap size) (elastic#128324) * WIP * Fix behavior when no plugins are defined * Remove unused import, reduce debounce times * Fix startup behavior * Misc improvements following PR comments * Fix plugin_status UTs * Code cleanup + enhancements * Remove fixed FIXME (cherry picked from commit a645545)

…ucing heap size) (#128324) (#129507) * Rewrite plugin_status logic limiting usage of Observables (reducing heap size) (#128324) * WIP * Fix behavior when no plugins are defined * Remove unused import, reduce debounce times * Fix startup behavior * Misc improvements following PR comments * Fix plugin_status UTs * Code cleanup + enhancements * Remove fixed FIXME (cherry picked from commit a645545) * Add extra tests from #128769 * Fix linting issues (older typescript version?) * Fix TS version issues

…eap size) (elastic#128324) * WIP * Fix behavior when no plugins are defined * Remove unused import, reduce debounce times * Fix startup behavior * Misc improvements following PR comments * Fix plugin_status UTs * Code cleanup + enhancements * Remove fixed FIXME

WIP

4b023f4

gsoldevila added Team:Core Platform Core services: plugins, logging, config, saved objects, http, ES client, i18n, etc t// release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting v8.2.0 labels Mar 22, 2022

gsoldevila requested a review from mshustov March 22, 2022 22:27

gsoldevila added 2 commits March 23, 2022 10:39

Fix behavior when no plugins are defined

02e19a8

Merge branch 'main' into kbn-128061-improve-plugin-status

6113711

mshustov added the ci:deploy-cloud label Mar 23, 2022

jbudz self-requested a review March 23, 2022 16:44

jbudz reviewed Mar 23, 2022

View reviewed changes

Remove unused import, reduce debounce times

6f3ac92

mshustov reviewed Mar 24, 2022

View reviewed changes

mshustov approved these changes Mar 24, 2022

View reviewed changes

gsoldevila added 4 commits March 24, 2022 13:13

Fix startup behavior

0ee3879

Misc improvements following PR comments

7c71a62

Merge branch 'main' into kbn-128061-improve-plugin-status

c1f15f3

Fix plugin_status UTs

3d64fed

gsoldevila marked this pull request as ready for review March 24, 2022 18:43

gsoldevila requested a review from a team as a code owner March 24, 2022 18:43

TinaHeiligers added loe:medium Medium Level of Effort impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. labels Mar 24, 2022

gsoldevila added 3 commits March 25, 2022 09:48

Code cleanup + enhancements

1301ded

Merge branch 'main' into kbn-128061-improve-plugin-status

f1dcea7

Remove fixed FIXME

9db8466

gsoldevila changed the title ~~WIP~~ Rewrite plugin_status logic limiting usage of Observables (reducing heap size) Mar 25, 2022

gsoldevila merged commit a645545 into elastic:main Mar 25, 2022

mshustov mentioned this pull request Mar 28, 2022

[Performance] Kibana RSS / Memory usage seems higher in 8.2 vs previous releases (by 50% sometimes) #128061

Closed

afharo mentioned this pull request Mar 29, 2022

remove deps on data plugin #121955

Closed

Dosant mentioned this pull request Mar 29, 2022

data dependent on security breaks tests #123720

Closed

This was referenced Mar 29, 2022

Update status when summary is updated #128769

Merged

Failing test: Jest Integration Tests.x-pack/plugins/fleet/server/integration_tests - Fleet preconfiguration reset Preconfigred cloud policy Works and preconfigure correctly agent policies #127075

Closed

gsoldevila mentioned this pull request Apr 5, 2022

[7.17] Rewrite plugin_status logic limiting usage of Observables (reducing heap size) (#128324) #129507

Merged

tylersmalley added ci:cloud-deploy Create or update a Cloud deployment and removed ci:deploy-cloud labels Aug 17, 2022

gsoldevila mentioned this pull request Jan 10, 2023

Fix unknown product test flaky behavior #148616

Merged

gsoldevila mentioned this pull request Oct 6, 2023

Add smart logic to log information about plugin status changes #168207

Merged

Conversation

gsoldevila commented Mar 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Attempt to solve #128061

Key changes to reduce heap size and improve performance

Memory consumption impact

Uh oh!

mshustov commented Mar 23, 2022

Uh oh!

mshustov commented Mar 23, 2022

Uh oh!

mshustov commented Mar 23, 2022

Uh oh!

jbudz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mshustov Mar 24, 2022

Choose a reason for hiding this comment

Uh oh!

gsoldevila Mar 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mshustov Mar 24, 2022

Choose a reason for hiding this comment

Uh oh!

gsoldevila Mar 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Mar 24, 2022

Uh oh!

kibana-ci commented Mar 25, 2022

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

gsoldevila commented Mar 22, 2022 •

edited

Loading

gsoldevila Mar 24, 2022 •

edited

Loading

gsoldevila Mar 24, 2022 •

edited

Loading