Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the call to on_off cluster in TuyaLevelControl #1748

Merged
merged 1 commit into from
Sep 12, 2022

Conversation

javicalle
Copy link
Collaborator

This PR will remove the call to on_off cluster when calling the move_to_level_with_on_off in the TuyaLevelControl (MCU) cluster.
The current quirk is completely wrong and needs to be fixed.

TL;DR;
But let me explain it from the begining...

In #1489 (ignoring some good practices) a call to the on_off cluster is added in the move_to_level_with_on_off command. This change is caused by a change in HA where dimmers now invoke this command (before the call from HA was different).
Some users report that the behavior of their devices has changed and that now its devices do not turn on when using the dimmer bar from HA.
The programmer in charge of #1489, misreading the arguments to the move_to_level_with_on_off command, (mistakenly) sets args[1] to the on/off value of the device.
He thought that invoking the on/off command for those devices that respected the standard couldn't hurt, and that on those that didn't it would be an improvement.
What could go wrong? Spoiler: args[1] is actually the transition_time:

This is the origin of the reported issues (or at least some of them).

Since the time this 'improvement' was implemented, ZHA has also evolved and now implements a mechanism to force the invocation of the on command for those devices that do not respect the standard:

With the new approach, the call is removed from the quirk because ZHA already have a mechanism to force the call to the on command.
Now the devices detected as problematic can be added in ZHA (just a single place).

This PR will remove the call to `on_off` cluster when calling the `move_to_level_with_on_off` in the `TuyaLevelControl` (MCU) cluster.
@javicalle
Copy link
Collaborator Author

But what will happen to the off command?
ZHA does not implement a solution for those devices that when setting the brightness to 0 do not turn off completely.

What is clear to me is that the current implementation of move_to_level_with_on_off is wrong and that this PR restores the 'normal' situation.
But I am also sure that more issues will arrive due to the change, in fact they will arrive regardless of the change since each user will expect a different behavior when change brightness in HA.

@coveralls
Copy link

Pull Request Test Coverage Report for Build 3039989687

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.03%) to 71.609%

Totals Coverage Status
Change from base Build 3037037987: 0.03%
Covered Lines: 5269
Relevant Lines: 7358

💛 - Coveralls

@dmulcahey
Copy link
Collaborator

Per the spec when using move_to_level_with_on_off 0 means turn the light off:

image

Is that not what was happening?

@javicalle
Copy link
Collaborator Author

I have no evidence, I bet you a lot of money that we have Tuya devices that don't do it and that stay bright at a minimum 🤷🏻‍♂️

@javicalle
Copy link
Collaborator Author

Note that I am addressing only Tuya devices with MCUs (very far from standard-compliant)

@dmulcahey
Copy link
Collaborator

I should have looked at the code change first. I completely agree with what you have changed here.

@dmulcahey
Copy link
Collaborator

You all done w/ this?

@javicalle
Copy link
Collaborator Author

Another approach could be to replicate all the brightness related treatment from the async_turn_on and make the calls to the on/offcommand, but some reports make me suspect that some devices have a 'toggle' behavior and the extra on command wil turn off the device

@javicalle
Copy link
Collaborator Author

Yes I think that is the right approach.
I'm ready to for the new issues 😄

@javicalle
Copy link
Collaborator Author

I should have looked at the code change first. I completely agree with what you have changed here.

The previous version was 🤦🏻‍♂️

@MattWestb
Copy link
Contributor

I dont remember but i think 0 in light level is not the same the the light is off 0x0006 can still being on.

Light level is little tricky and can have status then light is off and more strange things.

From
ZigBee Lighting & Occupancy
Device Specification
Version 1.0

1379 27.4.1.1 Attributes
1380 For devices implemented according to this specification, the CurrentLevel attribute SHALL be 
1381 interpreted as follows:
1382  A value of 0x00 SHALL not be used.
1383  A value of 0x01 SHALL indicate the minimum level that can be attained on a device.
1384  A value of 0xfe SHALL indicate the maximum level that can be attained on a device.
1385  A value of 0xff SHALL represent an undefined value.
1386  All other values are application specific gradations from the minimum to the maximum level.

@dmulcahey
Copy link
Collaborator

I dont remember but i think 0 in light level is not the same the the light is off 0x0006 can still being on.

Light level is little tricky and can have status then light is off and more strange things.

From ZigBee Lighting & Occupancy Device Specification Version 1.0

1379 27.4.1.1 Attributes
1380 For devices implemented according to this specification, the CurrentLevel attribute SHALL be 
1381 interpreted as follows:
1382  A value of 0x00 SHALL not be used.
1383  A value of 0x01 SHALL indicate the minimum level that can be attained on a device.
1384  A value of 0xfe SHALL indicate the maximum level that can be attained on a device.
1385  A value of 0xff SHALL represent an undefined value.
1386  All other values are application specific gradations from the minimum to the maximum level.

See this comment: #1748 (comment)

@dmulcahey dmulcahey merged commit a072ca3 into zigpy:dev Sep 12, 2022
@javicalle javicalle deleted the patch-3 branch September 12, 2022 19:44
@javicalle
Copy link
Collaborator Author

Later I review the open issues for this reason and report the change.

Thanks to both.

@MattWestb
Copy link
Contributor

Minimum level for the device = 1 or larger and not zero.
You must reading ZigBee Lighting & Occupancy as one appendix to 3 Zigbee spec or you is braking many light function in the system.

@dmulcahey
Copy link
Collaborator

Minimum level for the device = 1 or larger and not zero.
You must reading ZigBee Lighting & Occupancy as one appendix to 3 Zigbee spec or you is braking many light function in the system.

no, this is straight from zigbee cluster library specification. Also I am fairly certain that it's anything less than the minimum and not just === the minimum.

@dmulcahey
Copy link
Collaborator

dmulcahey commented Sep 12, 2022

https://zigbeealliance.org/wp-content/uploads/2019/12/07-5123-06-zigbee-cluster-library-specification.pdf

this is an older one and it is section 3.10.2.4.5 but it is consistent with the newest version from above.

@MattWestb
Copy link
Contributor

MattWestb commented Sep 12, 2022

OK from ZCL R8:
Cluster Library Specification
Document 07-5123 Revision 8
Chapter Document 14-0125
:

7565 3.19.2.2.1 CurrentLevel Attribute for Lighting
7566 A value of 0 SHALL not be used.
7567 A value of 1 SHALL indicate the minimum level that can be attained on a device.
7568 A value of 0xfe SHALL indicate the maximum level that can be attained on a device.
7569 A non-value of 0xff SHALL represent an undefined value.
7570 All other values are application specific gradations from the minimum to the maximum level.

@dmulcahey
Copy link
Collaborator

you are looking at attribute docs and not command docs. here is another from the same section in the level control cluster:

image

specifically:

"When the level is reduced to its minimum the OnOff attribute is automatically turned to Off, and when the level is increased above its minimum the OnOff attribute is automatically turned to On." I think this confirms my thoughts above that they don't have to match exactly and that simply surpassing the value is what triggers the change.

@MattWestb
Copy link
Contributor

That is true minimum = 1 if reading both then you have not red ZERO.

@dmulcahey
Copy link
Collaborator

dmulcahey commented Sep 12, 2022

image

min level is documented as accepting 0

@MattWestb
Copy link
Contributor

7540 3.19 Level Control for Lighting
7541 3.19.1 Overview
7542 Please see Chapter 2 for a general cluster overview defining cluster architecture, revision, classification, 
7543 identification, etc.
7544 This cluster provides an interface for controlling the level of a light source. 
7545 All requirements and dependencies are derived from the base cluster. Additional or extended requirements 
7546 and dependencies are listed in this cluster specification.
7547
7548 NOTE: This cluster specification is derived from the Level cluster specification (generic level control also 
7549 defined in this document). This cluster specifies further requirements for the lighting application. Please see 
7550 section 3.10 for the generic Level cluster specification.

@dmulcahey
Copy link
Collaborator

7540 3.19 Level Control for Lighting
7541 3.19.1 Overview
7542 Please see Chapter 2 for a general cluster overview defining cluster architecture, revision, classification,
7543 identification, etc.
7544 This cluster provides an interface for controlling the level of a light source.
7545 All requirements and dependencies are derived from the base cluster. Additional or extended requirements
7546 and dependencies are listed in this cluster specification.
7547
7548 NOTE: This cluster specification is derived from the Level cluster specification (generic level control also
7549 defined in this document). This cluster specifies further requirements for the lighting application. Please see
7550 section 3.10 for the generic Level cluster specification.

this is not ZCL or the ZHA profile specification this is in ZLO.

@MattWestb
Copy link
Contributor

Nop its the tales ZCL R8

@MattWestb
Copy link
Contributor

@dmulcahey
Copy link
Collaborator

it's a different cluster lol.

@MattWestb
Copy link
Contributor

7554 3.19.1.3 Cluster Identifiers
Identifier Name
0x0008 Level Control for Lighting

@MattWestb
Copy link
Contributor

I think all the logic is explained in:
7586 3.19.4 State Change Table for Lighting

@dmulcahey
Copy link
Collaborator

ok so now I am honestly confused. this overrides CurrentLevel definition of the base cluster but nothing else... say we follow this... all that changes is that in HA (ZHA) we would cap 0 to 1. The experienced functionality would not change. Are you saying if we in fact send 1 that the bulb would behave correctly?

@MattWestb
Copy link
Contributor

Read the posted min level on level cluster is allowed to have and and then the last posted 7586 3.19.4 State Change Table for Lighting.

Device min level is >1 or if sett in the device (but cant being less then 1) and the matrix is the logic that is being used if its one OK ZB3 device.

I only saying dont use Zero then its not allowed in ZB3 devices that is not to old.

If tuya is having 53629947 as min its in the end ZCL R7.5 that is being used in ZHA = larger then zero.

@dmulcahey
Copy link
Collaborator

dmulcahey commented Sep 12, 2022

I wonder... I bet this is related to your comments @MattWestb home-assistant/core#78206

These would all be changes for HA / ZHA and I think we are on the path to getting to an agreement in there... doesn't affect this though imo

@gonzalezjj
Copy link

I don't think this PR solves the underlying issue.

Instead of sending the command to the zigbee device, the quirk handles the command itself and directly controls the attributes of the device. Therefore, setting the on_off attribute when the move_to_level_with_on_off command is called is correct according to the ZCL spec. Otherwise, you're only ever setting the current_level attribute, which is independent of the on_off attribute, so the devices using this quirk will never turn on or off when the move_to_level_with_on_off command is sent.

The current implementation in the master branch is wrong, since it assumes args[1] is a boolean "on_off" parameter. But the solution should be to use args[0], the level, and set on_off according to that. With that implementation, the device would "correctly" handle the command (via its quirk), so ZHA would not need to add extra handlers.

@javicalle
Copy link
Collaborator Author

Instead of sending the command to the zigbee device, the quirk handles the command itself and directly controls the attributes of the device. Therefore, setting the on_off attribute when the move_to_level_with_on_off command is called is correct according to the ZCL spec. Otherwise, you're only ever setting the current_level attribute, which is independent of the on_off attribute, so the devices using this quirk will never turn on or off when the move_to_level_with_on_off command is sent.

As I've already said, ZHA offers a mechanism to work around that problem:

But the solution should be to use args[0], the level, and set on_off according to that.

I am not convinced on this point. It might seem like a simple solution, but we would duplicate the code in 2 places (ZHA and in the quirk). It may seem that it is a minor problem, but it is not, and you have the example in the conversation that @dmulcahey and @MattWestb have had.

There is a flaw in the handling (and interpretation) of the dimmers minLevel and maxLevel values.
And surely it will have to end up adapting sometime the implementation of the dimmer commands and the logic that you want to duplicate.
And I'm sure when that happens and both implementations are out of sync, there will be issues.
We are not enough to be aware of all the changes (we barely have enough to support the requests that arrive)

But... the main drawback that I see in this approach is that the solution involves adding the code of the manufacturer of each device in the ZHA definition:

@STRICT_MATCH(
    channel_names=CHANNEL_ON_OFF,
    aux_channels={CHANNEL_COLOR, CHANNEL_LEVEL},
    manufacturers={"Jasco", "Quotra-Vision", "eWeLight", "eWeLink"},
)
class ForceOnLight(Light):
    """Representation of a light which does not respect move_to_level_with_on_off."""

    _FORCE_ON = True

In the Tuya devices this is the TZE200_3p5ydos3 one.
And there are too many Tuya devices that will require that implementation. I think that every week 1 or 2 new devices of this type can come out.
And that's going to end up messing up the code in ZHA, which I don't like either (IMHO I prefer all of this to be inside the quirk)

And this is the dilemma that must be solved...

@gonzalezjj
Copy link

I understand your concerns, code duplication is never good. Since I'm fairly new to zigpy/zha-device-handlers/HA, I might be also be overlooking some aspects of the architecture that would break or add unnecessary overhead with my proposed approach. Nevertheless, I still believe that the correct place to fix this is in the quirk, and not in ZHA.

I am not convinced on this point. It might seem like a simple solution

While it is a simple solution, I wasn't proposing it due to its simplicity. The description of the zha-device-handlers project states that its function is to handle deviations from the ZCL standard in order to provide HA with a standard-compliant interface. In this case, the quirk exposes the move_to_level_with_on_off command, but it's not following the standard itself, since it's not doing anything about the _with_on_off part. If you work around that bug by changing HA, you're effectively adding another "quirks" layer to handle the issues in the zha-device-handlers code. I think the "quirks" functionality (i.e. code that works around ZCL spec violations) should be kept in a single layer, because every layer that works around issues of the one below is an additional component that you have to keep synchronized and will likely break if someone decides to tweak or fix the incorrect behavior.

There is a flaw in the handling (and interpretation) of the dimmers minLevel and maxLevel values.

I had that feeling, since my HA dimmer sliders will happily go down to 1% even though the dimmers do not go darker than the value set at around 20%. Unfortunately, I would not know where to start to fix this. At least one layer would have to fetch the minLevel and maxLevel from the zigbee device and cache those values in order to adapt what 1% and 100% mean.
For now, however, it seems that HA does not care about the actual minLevel and maxLevel values and implicitly assumes that minLevel is 0, so fixing this bug by setting on_off based on args[0] > 0 in the quirk feels like the correct approach.

@puddly
Copy link
Contributor

puddly commented Sep 13, 2022

@dmulcahey and I discussed yesterday what @MattWestb pointed about about the spec and I think the current behavior in ZHA (and the entire ForceOnLight class) may be due to a misunderstanding on our part about the interactions between min/max levels and move_to_level_with_on_off.

From what I gather:

  1. All move commands of the Level cluster must set the level to something between the min_level and the max_level, inclusive. However, the spec seems to indicate that there is a difference between move_to_level and move_to_level_with_on_off: sending move_to_level(level=min_level, transition=0) will set the bulb to its minimum brightness, but move_to_level_with_on_off(level=min_level, transition=0) will turn it off.

  2. ZCL R7/8 introduces a restriction of the Level cluster when it's used for a light and the cluster_revision attribute is 2: min_level is now always 0x01 and max_level is now 0xFE.

  3. Combining these two, we have a problem: move_to_level_with_on_off(level=0, transition=0) is not valid. Older devices seem to accept it, but the standards-compliant response should be INVALID_VALUE, which the eWeLight device is sending. We need to be sending move_to_level_with_on_off(level=min_value, transition=0) to turn the bulb off and then handle the special cases of turning a bulb on to its minimum value.

I'm not sure if our understanding of the spec is correct, but it seems to match up with what that eWeLight bulb is doing and how existing bulbs on our networks handle 0 and 1 as levels.

@MattWestb
Copy link
Contributor

MattWestb commented Sep 13, 2022

The implanted tuya MCU min and max dimmer is only reading and writing values and mapping them after conversation to ZHA Zigbee standard. If not making it 100% true is not one large problem if ZHA is tolerant for the 0 light level attribute (but shall not being that if being ZCL 7.5 compatible). But as i have saying better doing it right the first time or you is getting larger problems later (Z2M is using 256 as max and its 110% wrong).

With the "device min level" can being larger then 1 if the device have getting the attribute set to somthing else then standard so its making its more complicated if the device is having 25 sett and then need taking that in the calculation.

Great having our P online then doing ZCL talking !!!

Also ZLO was integrated in one of the latest ZCL reversions and i think ZHA have not being updated with that part to 100%.

@javicalle
Copy link
Collaborator Author

Too many paralel conversations here...

@gonzalezjj I must say that the ZHA approach I like it less and less. I would try to restore the previous behavior (this time well implemented)

I would leave the implementation of Tuya devices (Tuya MCU) for last. Let's see where this road takes us and then we'll worry about them.

All the min_level & max_level will have a lot of derivatives. Probably we can put all together in a discussion or another issue.

@dmulcahey
Copy link
Collaborator

With the "device min level" can being larger then 1 if the device have getting the attribute set to somthing else then standard so its making its more complicated if the device is having 25 sett and then need taking that in the calculation.

This is true but the way we understand what is in the spec is that 1 is a special value that means use the devices minimum value. Is that not how you are reading it? #1748 (comment)

“ A value of 1 SHALL indicate the minimum level that can be attained on a device.”

javicalle added a commit to javicalle/zha-device-handlers that referenced this pull request Sep 14, 2022
Implement the `on_off` call in the `move_to_level_with_on_off` command for the MCU Tuya devices.
The previous implementation was removed in zigpy#1748. Originally introduced in zigpy#1489
dmulcahey pushed a commit that referenced this pull request Sep 25, 2022
* Implement the `move_to_level_with_on_off` command 

Implement the `on_off` call in the `move_to_level_with_on_off` command for the MCU Tuya devices.
The previous implementation was removed in #1748. Originally introduced in #1489

* Fix black coverage
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants