Avoid returning statistics columns that the metadata knows are impossible

[bdraco]

Since we don't document the stats websocket apis, I'm not 100% sure we should publish this as a breaking change since that implies we expect the api to be stable and I don't think we actually have that guarentee. If we do want to do a breaking change for this, I'll do a blog post as well.

Breaking change

This breaking change is only for custom frontend developers who are using the statistics websocket apis.

statistics no longer returns columns of empty values when the metadata knows they are impossible. If has_sum is False for all queried statistics the sum and state columns will not be returned. If has_mean is False for all queried statistics, the mean, min, and max columns will not be returned.

This allows consumers of the api to avoid the overhead of populating empty columns (selecting an empty column has nearly the same overhead as a column with data) when they not aware (or cannot practically request) of the metadata in advance.

Proposed change

I tried to handle this on the frontend, but we don't have the metadata soon enough and would have to pay an additional round trip to get it. That solution didn't work out and is being reverted in home-assistant/frontend#16328 This replaces that.

We want to avoid searching the database for empty fields when the metadata already tells us they will be empty. This prevents generating larger responses with columns of the same empty field over and over.

Remove columns from the result when statistics_during_period notices the caller asks for an impossible request because all the requested stats do not have has_mean (mean/min/max columns) or has_sum (sum/state) columns

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Deprecation (breaking change to happen in the future)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue:
• Link to documentation pull request:

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• I have followed the perfect PR recommendations
• The code has been formatted using Black (black --fast homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
• Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[home-assistant]

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of recorder can trigger bot actions by commenting:

• @home-assistant close Closes the pull request.
• @home-assistant rename Awesome new title Renames the pull request.
• @home-assistant reopen Reopen the pull request.
• @home-assistant unassign recorder Removes the current integration label and assignees on the pull request, add the integration domain after the command.

[bdraco]

There are lots of tests that check for empty data that will need to be updated

[frenck]

I think this is fine. This API is not public (as in, not publicly supported/described); its sole purpose is powering the frontend.

I do think a dev blog post would help, but, we should be clear about the support in that case.

../Frenck

[bdraco]

Thanks