Live db migrations and recovery

[bdraco]

Breaking change

Run database migrations in the background after started event

• Database migrations no longer block startup
• The migration is started shortly after the started event since they are cpu intensive (mostly just sqlite since its in the same process) and we do not want to compete with startup.
• Events are queued and processed when the migration is completed
• There is a safety to start discarding events if more than ~30000 happen before migration is completed to avoid memory exhaustion
• A notification is shown when the migration is in progress "The database is being upgraded. Integrations such as logbook and history that read the database may return inconsistent results until the migration completes."
• The sqlite database recovery logic can now recover at point after setup including purge, migration, and event insert. In short we can always (hopefully) recover and start a new db without a restart.
• The quick check is no longer performed on unclean shutdown since we can always recover live. The db_integrity_check option has been deprecated.
• Recorder __init__.py coverage increased to 98% 🎊

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Example entry for configuration.yaml:

# Example configuration.yaml

Additional information

• This PR fixes or closes issue: fixes recorder starts reporting "database disk image is malformed" error #47936
• This PR is related to issue:
• Link to documentation pull request: Updates for live db migration and recovery home-assistant.io#17402

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• The code has been formatted using Black (black --fast homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

• No score or internal
• 🥈 Silver
• 🥇 Gold
• 🏆 Platinum

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[probot-home-assistant]

Hey there @home-assistant/core, mind taking a look at this pull request as its been labeled with an integration (history) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[bdraco]

100.00% of diff hit 🎉

[bdraco]

Pushed again to cover one more existing line. At this point testing has been solid

[bdraco]

Tested this on my large install with

drop index ix_events_event_type_time_fired;
drop index ix_events_context_parent_id;
delete from schema_changes where change_id>=2;

Restart

[bdraco]

More exhaustive test

update schema_changes set schema_version=0 where change_id=1;
delete from schema_changes where change_id>=2;
drop index ix_events_context_parent_id;
drop index ix_events_event_type_time_fired;
drop index ix_states_entity_id_last_updated;
drop index ix_states_event_id;
drop index ix_states_last_updated;
drop index ix_events_time_fired;

[bdraco]

I think we should warn that performance may be degraded during the upgrade so they don't wonder why.

[marvin-w]

Being one of the people that waited for about 25 hours during the last update this is a really awesome change! Well done!

[balloob]

Should we mention that they should not restart their Home Assistant installation?

We probably want to show this in the frontend somewhere too.

[bdraco]

Yes. That's a good idea 👍

[bdraco]

Maybe fire an event 'recorder_migration_started' 'recorder_migration_ended'. We could then show a warning on the server tools by the restart button that they probably shouldn't do it

[bdraco]

I wonder if shutdown will block actually since the thread won't exit until it finished. I'll test

[bdraco]

System performance will temporarily degrade during the database upgrade. Do not power down or restart the system until the upgrade completes. Integrations that read the database, such as logbook and history, may return inconsistent results until the upgrade completes.

Changed wording to ^

[bdraco]

The webserver will shutdown but the database migration continues and restart is blocked until it finishes because the thread won't join.

[bdraco]

2021-04-12 06:08:27 WARNING (MainThread) [homeassistant.core] Timed out waiting for shutdown stage 1 to complete, the shutdown will continue

[bdraco]

Shutdown blocks until the migration is complete because the thread can't join. I'm going to do another PR to check if migration is in progress and throw when calling SERVICE_HOMEASSISTANT_STOP or SERVICE_HOMEASSISTANT_RESTART so they don't end up in this state unexpectedly.

Thread 213 (idle): "MainThread"
    _wait_for_tstate_lock (threading.py:1027)
    join (threading.py:1011)
    shutdown (concurrent/futures/thread.py:236)
    close (homeassistant/runner.py:92)
    run (asyncio/runners.py:51)
    run (homeassistant/runner.py:126)
    main (homeassistant/__main__.py:313)
    <module> (homeassistant/__main__.py:321)
    _run_code (runpy.py:87)
    _run_module_as_main (runpy.py:194)
Thread 237 (idle): "SyncWorker_1"
    _wait_for_tstate_lock (threading.py:1027)
    join (threading.py:1011)
    shutdown (homeassistant/components/recorder/__init__.py:364)
    run (concurrent/futures/thread.py:57)
    _worker (concurrent/futures/thread.py:80)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 245 (idle): "Recorder"
    migrate_schema (homeassistant/components/recorder/migration.py:60)
    _migrate_schema_and_setup_run (homeassistant/components/recorder/__init__.py:523)
    run (homeassistant/components/recorder/__init__.py:435)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 251 (idle): "Thread-8"
    loop (paho/mqtt/client.py:1167)
    loop_forever (paho/mqtt/client.py:1779)
    _thread_main (paho/mqtt/client.py:3452)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 259 (idle): "Thread-9"
    loop (paho/mqtt/client.py:1167)
    loop_forever (paho/mqtt/client.py:1779)
    _thread_main (paho/mqtt/client.py:3452)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)

[bdraco]

Verified everything went down cleanly and restarted successfully once the migration did finish.

[bdraco]

Service block started here #49098

Need to think about how to tell the frontend as it needs to know on reload.

[balloob]

This looks great ! 🎉

Had a couple of minor comments.

[bdraco]

Thanks. Working on the shutdown block pr now