Skip to content

Always use UTC time for the MariaDB/MySQL session#87583

Merged
bdraco merged 6 commits into
home-assistant:devfrom
bdraco:mariadb_mysql_utc
Feb 9, 2023
Merged

Always use UTC time for the MariaDB/MySQL session#87583
bdraco merged 6 commits into
home-assistant:devfrom
bdraco:mariadb_mysql_utc

Conversation

@bdraco
Copy link
Copy Markdown
Member

@bdraco bdraco commented Feb 6, 2023 •
edited
Loading

Proposed change

Currently the MariaDB/MySQL session uses the default global time zone and we convert
all times to UTC via process_timestamp https://github.com/home-assistant/core/blob/dev/homeassistant/components/recorder/models.py#L85

This behavior leads to incorrect results during daylight savings time
since there are multiple possible times that resolve to the same UTC timestamp.

By setting the session up as UTC time we avoid the double conversion

  • the internal conversion when MariaDB retrieves the column to the global time zone
  • the process_timestamp conversion that was done in python to get the time zone back to UTC.

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • The code has been formatted using Black (black --fast homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
  • Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

@bdraco bdraco requested a review from a team as a code owner February 6, 2023 22:54
@home-assistant
Copy link
Copy Markdown
Contributor

home-assistant Bot commented Feb 6, 2023

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of recorder can trigger bot actions by commenting:

  • @home-assistant close Closes the issue.
  • @home-assistant rename Awesome new title Change the title of the issue.
  • @home-assistant reopen Reopen the issue.
  • @home-assistant unassign recorder Removes the current integration label and assignees on the issue, add the integration domain after the command.

@bdraco bdraco force-pushed the mariadb_mysql_utc branch from 6e18ea4 to 21a1944 Compare February 6, 2023 22:57
@bdraco bdraco mentioned this pull request Feb 6, 2023
Merged
24 tasks
@bdraco
Copy link
Copy Markdown
Member Author

bdraco commented Feb 7, 2023

I wanted to tag this for a patch release as I think it will solve some unexplained issues but I'm a bit nervous about it as I don't have a large enough test surface to feel 100% on it.

@bdraco
Copy link
Copy Markdown
Member Author

bdraco commented Feb 8, 2023

Uncovered lines are the postgresql and mysql test runners which don't currently report to codecov

@bdraco bdraco mentioned this pull request Feb 8, 2023
Merged
19 tasks
@epenet
Copy link
Copy Markdown
Contributor

epenet commented Feb 8, 2023

According to coverage postgre and mariadb should upload their coverage

[2023-02-08T15:03:52.923Z] ['info'] => Found 12 possible coverage files:
  coverage-3.10-1/coverage.xml
  coverage-3.10-10/coverage.xml
  coverage-3.10-2/coverage.xml
  coverage-3.10-3/coverage.xml
  coverage-3.10-4/coverage.xml
  coverage-3.10-5/coverage.xml
  coverage-3.10-6/coverage.xml
  coverage-3.10-7/coverage.xml
  coverage-3.10-8/coverage.xml
  coverage-3.10-9/coverage.xml
  coverage-3.10-mariadb/coverage.xml
  coverage-3.10-postgresql/coverage.xml
[2023-02-08T15:03:52.924Z] ['info'] Processing /home/runner/work/core/core/coverage-3.10-1/coverage.xml...
[2023-02-08T15:03:52.9[41](https://github.com/home-assistant/core/actions/runs/4126332885/jobs/7128682938#step:4:42)Z] ['info'] Processing /home/runner/work/core/core/coverage-3.10-10/coverage.xml...
[2023-02-08T15:03:52.963Z] ['info'] Processing /home/runner/work/core/core/coverage-3.10-2/coverage.xml...
[2023-02-08T15:03:52.979Z] ['info'] Processing /home/runner/work/core/core/coverage-3.10-3/coverage.xml...
[2023-02-08T15:03:53.005Z] ['info'] Processing /home/runner/work/core/core/coverage-3.10-4/coverage.xml...
[2023-02-08T15:03:53.020Z] ['info'] Processing /home/runner/work/core/core/coverage-3.10-5/coverage.xml...
[2023-02-08T15:03:53.0[48](https://github.com/home-assistant/core/actions/runs/4126332885/jobs/7128682938#step:4:49)Z] ['info'] Processing /home/runner/work/core/core/coverage-3.10-6/coverage.xml...
[2023-02-08T15:03:[53](https://github.com/home-assistant/core/actions/runs/4126332885/jobs/7128682938#step:4:54).063Z] ['info'] Processing /home/runner/work/core/core/coverage-3.10-7/coverage.xml...
[2023-02-08T15:03:53.086Z] ['info'] Processing /home/runner/work/core/core/coverage-3.10-8/coverage.xml...
[2023-02-08T15:03:53.101Z] ['info'] Processing /home/runner/work/core/core/coverage-3.10-9/coverage.xml...
[2023-02-08T15:03:53.125Z] ['info'] Processing /home/runner/work/core/core/coverage-3.10-mariadb/coverage.xml...
[2023-02-08T15:03:53.126Z] ['info'] Processing /home/runner/work/core/core/coverage-3.10-postgresql/coverage.xml...

@bdraco
Copy link
Copy Markdown
Member Author

bdraco commented Feb 8, 2023 •
edited
Loading

Something isn't seeing them because its running the same test on sqlite / mariadb / postgresql and those lines are covered for sqlite but its reporting them as covered for mariadb/postgresql. I'm not sure whats going on but hopefully we can sort it out in #87640

@epenet
Copy link
Copy Markdown
Contributor

epenet commented Feb 8, 2023 •
edited
Loading

I'm pretty sure it's a temporary error from codecov.
Sometimes it gets out of sync and takes a few (hours? commits? merges?) before it cleans up its act.

I don't think #87640 will change anything here.

@emontnemery emontnemery closed this Feb 9, 2023
@emontnemery emontnemery reopened this Feb 9, 2023
@emontnemery
Copy link
Copy Markdown
Contributor

emontnemery commented Feb 9, 2023

I'm pretty sure it's a temporary error from codecov. Sometimes it gets out of sync and takes a few (hours? commits? merges?) before it cleans up its act.

I don't think #87640 will change anything here.

That's not the case, the migration tests were hard coded to use an sqlite database, that was fixed by #87753

emontnemery
emontnemery approved these changes Feb 9, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@emontnemery emontnemery left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @bdraco 👍

@emontnemery emontnemery closed this Feb 9, 2023
@emontnemery emontnemery reopened this Feb 9, 2023
@bdraco bdraco merged commit db503c0 into home-assistant:dev Feb 9, 2023
@bdraco bdraco deleted the mariadb_mysql_utc branch February 9, 2023 17:39
AlePerla pushed a commit to AlePerla/homeassistant_core that referenced this pull request Feb 10, 2023
@bdraco @AlePerla
Always use UTC time for the MariaDB/MySQL session (home-assistant#87583)
b305526
@github-actions github-actions Bot locked and limited conversation to collaborators Feb 10, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@emontnemery emontnemery emontnemery approved these changes

Assignees

No one assigned

Labels

bugfix cla-signed core database: mariadb database: mysql has-tests integration: recorder Quality Scale: internal small-pr PRs with less than 30 lines.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@bdraco @epenet @emontnemery