Prevent sqlalchemy from refetching the old_state_id as it will never change#40982
Merged
balloob merged 1 commit intohome-assistant:devfrom Oct 1, 2020
Merged
Conversation
…change Disable expire_on_commit for the event writer. Since we never expect the old_state_id to change in the database, it was never worth the expense of refetching the id after the commit.
probot-home-assistant
Bot
added
Hacktoberfest
core
integration: recorder
small-pr
balloob
approved these changes
Oct 1, 2020
balloob
pushed a commit
that referenced
this pull request
Oct 1, 2020
…change (#40982) Disable expire_on_commit for the event writer. Since we never expect the old_state_id to change in the database, it was never worth the expense of refetching the id after the commit.
bdraco
added a commit
to bdraco/home-assistant
that referenced
this pull request
Oct 7, 2020
While home-assistant#40982 solved the performance overhead of expiring every time we commit the event session, it caused a regression which was fixed in home-assistant#41349. Ideally we could avoid the overhead of expiring objects on commit since we are never going to use them again by using expunge after the commit to remove the objects we no longer need. Unfortunately this causes sqlalchemy to spend quite a bit of time sorted out state when adding new objects to the event session after an expunge_all so it wasn't a viable option. As not expring the objects causes unexpected side effects, we can limit the impact of having to expring them by only doing the expire every 120 commits (by default every 2 minutes) instead of every commit (by default every second). This works because the expire operation itself is expensive reguardless of the number of objects it is expiring. To verify this approach does not leak memory, a scale test integration was created (https://github.com/bdraco/scaletest) and run for hours along with an objgraph dump setup every 30 seconds to verify States and Events were being properly disposed of (https://mg.pov.lt/objgraph/objgraph.html#locating-and-filtering-objects) In testing this took a machine running the scaletest integration from 100% cpu to 4% cpu
21 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed change
Prevent sqlalchemy from refetching the old_state_id as it will never change
Disable expire_on_commit for the event writer. Since we never expect the
old_state_id to change in the database, it was never worth the expense of
refetching the id after the commit.
I had thought I solved this in #40467 but
since I didn't have the number of state changes needed for it to come up in the profile,
I missed that sqlalchemy would refresh the old state id since it would expire the object
on commit (default behavior) even though it never changes.
Type of change
Example entry for
configuration.yaml:# Example configuration.yamlAdditional information
Checklist
black --fast homeassistant tests)If user exposed functionality or configuration variables are added/changed:
If the code communicates with devices, web services, or third-party tools:
Updated and included derived files by running:
python3 -m script.hassfest.requirements_all.txt.Updated by running
python3 -m script.gen_requirements_all..coveragerc.The integration reached or maintains the following Integration Quality Scale:
To help with the load of incoming pull requests: