Reduce orm overhead by grouping object expiration#41394
Merged
balloob merged 1 commit intohome-assistant:devfrom Oct 7, 2020
Merged
Reduce orm overhead by grouping object expiration#41394balloob merged 1 commit intohome-assistant:devfrom
balloob merged 1 commit intohome-assistant:devfrom
Conversation
While home-assistant#40982 solved the performance overhead of expiring every time we commit the event session, it caused a regression which was fixed in home-assistant#41349. Ideally we could avoid the overhead of expiring objects on commit since we are never going to use them again by using expunge after the commit to remove the objects we no longer need. Unfortunately this causes sqlalchemy to spend quite a bit of time sorted out state when adding new objects to the event session after an expunge_all so it wasn't a viable option. As not expring the objects causes unexpected side effects, we can limit the impact of having to expring them by only doing the expire every 120 commits (by default every 2 minutes) instead of every commit (by default every second). This works because the expire operation itself is expensive reguardless of the number of objects it is expiring. To verify this approach does not leak memory, a scale test integration was created (https://github.com/bdraco/scaletest) and run for hours along with an objgraph dump setup every 30 seconds to verify States and Events were being properly disposed of (https://mg.pov.lt/objgraph/objgraph.html#locating-and-filtering-objects) In testing this took a machine running the scaletest integration from 100% cpu to 4% cpu
balloob
approved these changes
Oct 7, 2020
bdraco
added a commit
to bdraco/home-assistant
that referenced
this pull request
Oct 7, 2020
…tant#41394)" This reverts commit 113d738.
bdraco
added a commit
to bdraco/home-assistant
that referenced
this pull request
Oct 7, 2020
…me-assistant#41394)"" This reverts commit ddfaed9.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed change
While #40982 solved the performance overhead of expiring every time
we commit the event session, it caused a regression which was
fixed in #41349.
Ideally we could avoid the overhead of expiring objects on commit
since we are never going to use them again by using expunge after
the commit to remove the objects we no longer need. Unfortunately
this causes sqlalchemy to spend quite a bit of time sorting out
state when adding new objects to the event session after an
expunge_all so it wasn't a viable option.
As not expring the objects causes unexpected side effects, we can
limit the impact of having to expring them by only doing the expire
every 120 commits (by default every 2 minutes) instead of every commit
(by default every second). This works because the expire operation
itself is expensive reguardless of the number of objects it is expiring.
To verify this approach does not leak memory, a scale test integration
was created (https://github.com/bdraco/scaletest) and run for hours
along with an objgraph dump setup every 30 seconds to verify States
and Events were being properly disposed of
(https://mg.pov.lt/objgraph/objgraph.html#locating-and-filtering-objects)
In testing this took a machine running the scaletest integration
from 100% cpu to 4% cpu
Type of change
Example entry for
configuration.yaml:# Example configuration.yamlAdditional information
Checklist
black --fast homeassistant tests)If user exposed functionality or configuration variables are added/changed:
If the code communicates with devices, web services, or third-party tools:
Updated and included derived files by running:
python3 -m script.hassfest.requirements_all.txt.Updated by running
python3 -m script.gen_requirements_all..coveragerc.The integration reached or maintains the following Integration Quality Scale:
To help with the load of incoming pull requests: