[TESTS] End-to-end tests

[ivmarkov]

I have the not so comfy feeling, that the tests covering rs-matter are just not enough, and so we are prone to regressions as we develop it over time. (Guilty as charged here, btw.)

Unit tests are one thing, but they can only go so far.
Also, the higher up you climb the stack, the more difficult it becomes to create and maintain such ones.

Though - I must say - the way I've influenced the existing semi-unit-tests/semi-e2e-tests that originally tested the Interaction/Data Layer is that they are currently more of an e2e test suite - as they are exercising aspects of the transport layer too - than what they were originally - tests bound to the Data layer only. So we might evolve this "framework" too, but writing the test cases themselves is super-boring, ergo, I'm looking at reusing someone else's test cases.

Regardless, and without digressing any further, the question:
@kedars, @andreilitvin: Are there / is there an e2e test framework in the C++ SDK that we can somehow extract and re-use in the context of rs-matter?

I'm imagining - roughly speaking - a bunch of automation tests - likely in the form of shell or Python scripts exercising the chip-tool command line and executed against an up-and-running version of the C++ SDK (which we should be able to replace with an up-and-running rs-matter stack).

So, does such an animal exist, in there? Can you point me to it?

[ivmarkov]

Hm, maybe this one?

[andreilitvin]

We have 2 sides: - yaml tests which are more simple read/validate and they also expose some virtual clusters for provisioning or timing. There is a python parser for those YAML tests - full python-driven tests In theory, a full python driven test (or any test for that matter) should be able to run against any app running on linux, including a rust application. See the app argument in https://github.com/project-chip/connectedhomeip/blob/master/.github/workflows/tests.yaml#L478 It may be useful to create example apps in rust and test them against the C++ test suite.

…

--Andrei

On Thu, Jun 13, 2024 at 3:54 AM ivmarkov ***@***.***> wrote: Hm, maybe this one <https://github.com/project-chip/connectedhomeip/tree/master/src/python_testing> ? — Reply to this email directly, view it on GitHub <#187 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB26UOD6TKXLHZC7O5NNT6DZHFF37AVCNFSM6AAAAABJHYNX7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRUHEYDAMJWGQ> . You are receiving this because you were mentioned.Message ID: <project-chip/rs-matter/issues/187/2164900164 ***@***.*** com>