Fuzz Test

[gwbres]

Mode / Dimensions / Signals

This library supports RTK + PPP with 3 measurement techniques, that means a total of 6 different modes as of today.
Eventually, a secondary solver will appear, dedicated to Dynamics prediction, to that doubles everything once again.
The number of degrees of freedom (NDF) differs from any other scenarios in RTK.
The NDF is more complex to handle in any PPP measurement technique, as it is different for each measurement.

We should have some sort of Fuzz test that is able to feed our CI/CD data but varying the measurements being fed, from epoch to epoch, especially:

• the number of satellites in sight
• the technique being used
• possibly the signal conditioning

The Fuzz infra should use the existing CI/CD data, which is in CI/CD by itself (due to the complexity of all these topics..), and is therefore validated, and is physically meaningful. So it should load this "nominal data" and possibly add variations of the NDF from there.

[gwbres]

src/tests/fuzz is already initiated but in very early stage.
It is totally possible that it is not the right approach and there are better things to do.
It requires more work in any case.
The most important point is to rely on the built-in test data points, and start from there.

[gwbres]

@ChristopherRabotin,

considering your experience with ANISE / Hifitime, you may have meaningful input.
What are your thoughts, considering that, in order to obtain correct test-results (so the filter does not diverge), we need to use the test-data (symbols) that are integerated within src/tests/data, we cannot use unrealistic and totally random data. But we want the dimensions to vary. In other words, we want to randomly drop a few points here & there, drop a satellite here & there. In the long term, this would make the library robust. Do you think one of the Fuzz infra that you have already used in the past, would fit the requirement to add variations in the degrees of freedom of our own test data

[ChristopherRabotin]

Of my libraries, only ANISE has fuzz testing, developed by SwitchbackEngineer. One of the issues is that fuzz testing takes in theory an infinite time to run. Google's oss-fuzz tries to mitigate this somehow, but it still takes a very long time to complete.

Hifitime as you know uses the formal model checker instead of a fuzz testing: this ensures that the inputs are within a specific range. Nyx however only uses rstest with different test scenarios: it neither has fuzz testing nor formal verification.

In my view, fuzz testing is important when the input data may be garbage (like in ANISE with the binary files). Formal verification is great if you want to test all possible valid bounds of a program and make sure it returns an error and does not panic. But for scenario based testing, rstest is perfect: similar dynamics but different input state, for example.

[gwbres]

Thank you very much,

it sounds like I need a combination of formal checking, because the inputs can only have specific values (otherwise it would diverge), and rstest that I already use but barely looked into it. This comforts me in the idea of creating my own infra to be able to iterate randomly and inject a little bit of randomness in the data. src/tests/fuzz (idle as of today)