delta filter - behaviour (just checking)

[luissimoes]

I have some inconsistent results when using a statistic of type sum and then using a filter: delta.

Just to be sure:
the documentation says: delta # computes the delta between each two consecutive numeric y values.

Does it provide also negative values to the sum? or just the positive values

[dbuezas]

It just the previous value from the last.
https://github.com/dbuezas/lovelace-plotly-graph-card/blob/master/src/filters/filters.ts#L111

Can you share your yaml and a screenshot of both the original statistic (without filters) and with it?

[luissimoes]

Here is what I have.... I dont understand what is going on, with the measurement (all details hereafter).

Situation
I measure the solar_energy_consumed through a sensor built on a template: I subtract the to_netz (energy sold) from the solar_energy_produced.

To generate the graph I use statistic: sum, period: day, filter: delta

  - entity: sensor.solar_energy_consumed
    texttemplate: '%{y:.2f}'
    textposition: outside
    textfont:
      size: 12
    name: solar consumed
    type: bar
    statistic: sum
    period: day
    marker:
      color: '#97cd07'
      opacity: 0.9
    filters:
      - delta
      - filter: i>0 & i < xs.length - 1
      - fn: |
          ({ys,xs,hass, vars}) => ({
            xs: [...xs, +xs.at(-1)+1000*60*60*24],
            ys: [...ys, hass.states['sensor.solar_energy_consumed'].state],
          })

The result is in the figure (includes the separate graph for solar energy produced)

Problems
There are clearly wrong values for the solar_energy_consumed in the graph, although the individual values of the sensor are correct

2024-02-14: should be 3.39 - is 7.08
2024-02-16: should be 2.32 - is: 6.16

--> a possible problem is off the "peaks" .... but that is why I dont understand how the stats work i.e. count

On the peaks
They are created by 2 reasons: (pictures at the end to improve readability)
-the ones during the day is because the information sent by the sensors sometimes is not available; I defaulted it by making the value 0 (zero) if it has no value

• the ones at 00:00 is because the sensor for to_netz resets exactly at 00:00 and the sensor for solar_energy_produce a few seconds (10?) later.... so the solar_energy_consumed at the start of the new day is still the same as at 00:00

My Questions and (not-)Conclusions
this was the reason for my post: how is the delta calculated?

• if the delta counts also "negative" increments (which is apparently does) then the peaks in the middle of the day have very little influence because the state before and after the stabilisation is the same (the ups are compensated by the downs and results in a net zero change)

the behaviours at 00:00 are puzzling

• the day starts with a value V that then goes (after 10 seconds) to 0..... so there is a -V delta that should be added by the statistics
• if the stats decide that (because the period is day) the day starts with zero, then there is a delta of V (the persistent value) and a delta of -V... and so the net value is zero....
• in this case there should also be no impact in the final value... ---> but the values are wrong and impacted! and it seems indeed that the reason is this peak at 00:00....

---

Add-Ons

template for solar_energy_consumed

{% set solar_prod = (states('sensor.solar_energy_daily_yield') |float(0)/1000) %}
{% set energy_sold = states('sensor.to_netz_daily')|float(0) %}
{% set solar_consumed = (solar_prod - energy_sold) | float(0) %}

{% if solar_consumed <= 0 %}
  {{ 0 }}
{% else %}
  {{ solar_consumed }}
{% endif %}

Pictures for the details of the peaks

1. middle of the day
   As can be seen the sensor is not available and as a result the sensor used in the template has a value of zero, and so the the solar_energy_consumed is zero...
   Later the sensor for the to_netz is briefly not available and as a result (it is read as zero) the solar_energy_consumed is counted as everything the was produced....

2. midnight
   Here the interesting case
   The sensor to_netz resets at 00:00 and the other seconds later... so the solar_energy_consumed at the start of the new day is still the same as at 00:00

[luissimoes]

Hi,
Here is the graphs I prepared: I have 2 sensors:

• sensor.pv_dailyyield: comes directly from the photovoltaic installation, and sometimes is the sensor that is unavailable)
• sensor.solar_daily_energy: its from the utility.yaml... and counts the energy from the above sensor daily i.e. it resets at 00:00

My conclusion for now:
"delta": resets at midnight i.e. doesn't count the value that is "brought" over at the time
Meaning:

• if the value at 00:00:00 was eg 5 and then at 00:05 is 0, I would have expected "delta" to show a value of -5
• from the example it seems that it counts from 0 to 5 (+5) and then from 5 to 0 (-5) resulting in a net zero value.

Note: for me this is OK and perfect. But the behaviour was not clear and I wanted to clear because it can affect calculations a lot.

The data

YAML

type: custom:plotly-graph
entities:
  - entity: sensor.pv_dailyyield
    type: bar
    statistic: sum
    period: day
    yaxis: y1
    marker:
      color: cyan
      opacity: 0.8
    filters:
      - delta
      - map_y_numbers: y/1000
  - entity: sensor.solar_daily_energy
    type: bar
    statistic: sum
    period: day
    yaxis: y1
    marker:
      color: orange
      opacity: 0.8
    filters:
      - delta
      - map_y_numbers: y/1000
hours_to_show: 72
refresh_interval: 10
title: All inputs

Now lets get the graph for the 2 sensors.
As can be seen the totals are exactly the same - no difference in adding.

Let's look at the details at midnight (00:00) for instance for the 18. to 19. (it's the same whatever the day).

---> the sensor.pv_dailyyield has a "holdover" onto the next day...

==== Delta Behaviour

1. if delta doesn't reset then the first change is a Negative value (around 2.8) and so the values of the graph could never be the same as shown
2. the values are the same... so this means that in this statistic, the delta "resets" at 00:00 and then as a positive and a negative increment, totalling net zero

---

For the sensor.pv_dailyyield

For the sensor.solar_daily_energy

[dbuezas]

I think the issue is that these are sum daily sensors. We only get 1 datapoint per day.
Sorry I can't go really deep analysing the problem at the moment, but I suggest plot the raw entities (without the delta filter) too. This way you can see the input to the delta and debug this better.
To deal with the reset issue, you could try period: 5minute and use a resample filter. This way you get more datapoints per day and can do something about the sensor resetting too early and breaking the total.

[luissimoes]

Thanks for the answers and helping

Actually the sensors update a few times per minute - while there is sun :-) - and they are energy increasing so they always have a value.

I solved using state and a sensor that resets at 00:00 defined in the utility yaml.

My point was to understand the delta behavior and seems it resets at 00:00 - that’s why there isn’t a negative delta in the final sum

[dbuezas]

The delta filter just subtracts the previous from each datapoint. Are you sure you are not confusing statistics with history?
Statistics is a different db in home assistant, and when you ask for daily statistics of a sensor, you only get 1 datapoint per day. If that statistic is configured to reset each day then that happens way before this gets into this card.

Try plotting this:

type: custom:plotly-graph
entities:
  - entity: sensor.pv_dailyyield # history db, not statistic db. Typically only holds one week of data.
  - entity: sensor.pv_dailyyield # statistics db, holds data indefinitely 
    statistic: sum
    period: day
  - entity: sensor.pv_dailyyield
    statistic: sum
    period: hour
  - entity: sensor.pv_dailyyield  # also statistics db but granlar, isn't kept forever
    statistic: sum
    period: 5minute

This will show you exactly what data comes in.

[luissimoes]

AH! that's clear now!!! I was not aware that one would get one value for the period ... makes a lot of sense!
And the plot you proposed is amazing to clarify! That you!
I plotted it and also added sum and delta... just to see it....

I see now .... (there is light at the end of the tunnel) :-)

THANKS!
All clear now!

what has always made me confused was the behaviour of the sensor that holds the value 20 seconds or so after midnight....
(see below - this is the same pv_dailyyield)

[dbuezas]

🎉 Great! At first I didn't understand what your issue was and then it clicked :)
Happy to hear this helped.

Regarding the late update, it may well be that the sensor updates every once a while, and some seconds after midnight just happens to be the first update of the day.

[dbuezas]

BTW, if you want to use the daily statistic but also have a bar with the current partial data of today, see this #299 (reply in thread)

[luissimoes]

thanks!
I am using now statistic: state, period: day (since it brings one point as you mentioned and it is the real value at the end of the day)... so its perfect!

I got a graph like the one below.. which is what I wanted to show all the components.

But regarding your suggestion....
your suggestion is (for the first filter)

Alternative 1)

    filters:
      - filter: i>0 && i < xs.length - 1
      - fn: |
          ({ys,xs,hass}) => ({
            xs: [...xs, new Date().setHours(0,0,0)],
            ys: [...ys, hass.states['sensor.from_netz_daily'].state],
          })

and I had (Alternative 2)

    filters:
      - filter: i>0 && i < xs.length - 1
      - fn: |
          ({ys,xs,hass, vars}) => ({
            xs: [...xs, +xs.at(-1)+1000*60*60*24],
            ys: [...ys, hass.states['sensor.from_netz_daily'].state],
          })

I guess it was from the tentatives to have the graph center values since I am getting the last value and adding a day

===> what is best? the first of the second? from a system point of view?
or is it the same?

thanks