Keep track of the derivative for unit_time

[basnijholt]

Breaking change

The derivative component is still in beta.

Proposed change

With this implementation, we will get a better estimate of the derivate during the timescale that is relevant to the sensor (indicated by the new option time_window).

This solves a problem where sensors have a low output resolution.
For example, a temperature sensor that can only report integer numbers (or half-integer).
It might report values every few seconds that are the same (e.g. 18 deg) and then suddenly go up one value (to 19 deg).
Only at that moment (with the current implementation) the derivative will be finite, otherwise, it will be zero. This instantaneous derivative isn't a good estimate of the true derivative.

The problem is captured by the following plot:

import matplotlib.pyplot as plt
import numpy as np
xs = np.linspace(0, 2 * np.pi)
true_sensor = 4 * np.sin(xs)
sensor_output = np.round(true_sensor, 0)
plt.plot(xs, true_sensor, label='true sensor')
plt.plot(xs, sensor_output, label='sensor value')
plt.scatter(xs, sensor_output, c="C1")
plt.plot(xs, np.gradient(sensor_output), label='calculated derivative (with current method)')
plt.plot(xs, 5*np.cos(xs), label='true derivative')
plt.legend()

With my proposed implementation, this problem will not occur, because it takes the average derivative of time_window.

I have added a time_window option, if you leave it out or if you set it to 0, the behavior is as it is in the dev branch.

Config example:

sensor:
  - platform: derivative
    source: sensor.gas_consumption
    name: Gas per hour
    unit: m³/h
    unit_time: h
    time_window: "00:30:00"

will average over the last half hour.

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Example entry for configuration.yaml:

# Example configuration.yaml

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue:
• Link to documentation pull request: Derivative component update, add time_window home-assistant.io#11963

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• The code has been formatted using Black (black --fast homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for Derivative component update, add time_window home-assistant.io#11963
  If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.

• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.

• Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

• No score or internal
• 🥈 Silver
• 🥇 Gold
• 🏆 Platinum

[probot-home-assistant]

Hey there @afaucogney, mind taking a look at this pull request as its been labeled with a integration (derivative) you are listed as a codeowner for? Thanks!

[afaucogney]

With this implementation, we will get a better estimate of the derivate during the timescale that is relevant to the sensor (indicated by unit_time).

unit_time is not about relevance, this is a part of the unit of the derivative result, that you can calculate for different time scale.

[basnijholt]

I've also editted the body of the first post in this PR.

[balloob]

Don't forget to add a test.

[basnijholt]

@balloob, I have added a test! 👍

I've also refactored the tests and found an actual bug (division instead of multiplication of unit_time) and fixed it (see this commit 243d75c).

There was a test that confirmed the broken state, which I've fixed in 3f9fcac.

There was also a bug in the UNIT_PREFIXES, where G was 1e6 instead of 1e9. In addition, I have added some more common prefixes.

[springstan]

Looks good to me, especially the tests look pretty clean now 👌🎉

[afaucogney]

@basnijholt, I'm happy you deeply reviewed the derivative integration. I'm neither a "python guy", neither a "derivative guy".

However I still have some open question:
1/ First I update you plot with same proportional factor for both derivatives (to get the same plot)

=> it looks better to me, and will provide less confusion. (with the first chart, it seems the original is almost always zero +- something)

• First, it would be good to show you result, (that we can compare with original version)
• Then, there is something I do not understand. You may see that the gradient of the sensor_value is high (always higher of the true_sensor if > 0) or 0.

=> From this fact, how the true_derivative can be higher (in value) than the calculated one (the original one) ?

PS: As I said at the beginning of the "derivative" PR, this code is based on the "integration" integration. So maybe you should have a look on it, as you discovered many findings here !

[afaucogney]

I played a bit with jupyter, then this should more realistic to me.

[basnijholt]

Hi @afaucogney, you are right! I made a mistake in my plot. I both used 5*np.cos(xs) instead of 4*np.cos(xs), so the derivative shouldn't be larger than the function itself.

Also, as you have correctly done, in my plot I didn't divide by Δx.

Your analysis shows, that if you have a densely sampled discrete sensor, that the derivate gives high peaks (which will become delta peaks in the limit of Δx → 0). That is the exact problem that I hoped to solve :)

I am doing some more small checks now.

Since now we essentially do a moving average, the derivative that we calculate is actually of t_now - time_window/2. I think we can still easily correct that too though.