Compare results with tolerance #390
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As an exercise I've re-implemented gunnarmorling#375 idea to compare result numbers with a tolerance: ``` $ ./test.sh baseline 0 src/test/resources/samples/measurements-rounding-precise.txt Validating calculate_average_baseline.sh -- src/test/resources/samples/measurements-rounding-precise.txt Rounding=14.6/25.5/33.6 != Rounding=14.6/25.4/33.6 (avg) ``` The hassle of parsing highlights the importance of using machine-readable format as suggested in gunnarmorling#14 **But** I think this approach is wrong, the rules should define rounding mode and baseline should be fixed to produce the correct result. Submissions should be fixed accordingly or qualified as non-passing. Could have been done earlier but better late than never. Updates gunnarmorling#49
|
Hey @AlexanderYastrebov, yeah, agreed that the "compare with tolerance" approach isn't the best one. I have opened #392 which fixes the baseline and adds a test case to ensure the correct behavior. Note I am not going to re-evaluate existing entries, compliance at the time of original evaluation is what decides (it's a common strategy, for instance also used in JCP when certifying spec implementations). New or updated entries must be compliant with the fixed behavior, by passing the additional test. |
|
So do we still need this in the light of the above, @AlexanderYastrebov? |
No, this was just an exercise. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As an exercise I've re-implemented #375 idea to compare result numbers with a tolerance:
The hassle of parsing highlights the importance of using machine-readable format as suggested in #14
But I think this approach is wrong, the rules should define rounding mode and baseline should be fixed to produce the correct result.
Submissions should be fixed accordingly or qualified as non-passing.
Could have been done earlier but better late than never.
Updates #49