Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add mutation testing to CI #156

Merged
merged 14 commits into from
Jan 15, 2024
Merged

Add mutation testing to CI #156

merged 14 commits into from
Jan 15, 2024

Conversation

tomato42
Copy link
Member

@tomato42 tomato42 commented Nov 2, 2019

related to #131

0.14.1: With just 20 mutants executed in 25 minutes, it's not really useful (the confidence interval is way too large)
0.15-beta (8deb089): Single test suite execution takes about 9.97s on Travis, in 25 minutes we execute 107 tests, so better, but that still translates to a 99.9% confidence interval of 5.39% to 26.38% for survival rate. So we will need a faster test suite still.
0.16.0: No significant change, about 152 tests in 25 minutes,

todo:

  • implement @slow decorator (pytest docs)
  • change hypothesis settings when option to run slow tests is enabled to execute fewer examples
  • mark all the slow tests, including hypothesis tests, that take more than few miliseconds to execute, with the @slow decorator
  • create new static test cases so that branch coverage doesn't fall with the slow tests disabled
  • implement small curve (Add support for 112, 128, and 160 curves #24, Add support for small curves #223), use it, as it's faster, to test correctness of the implementation instead of P-256, P-224 or P-192

@tomato42 tomato42 added the maintenance issues related to making the project usable or testable label Nov 2, 2019
@tomato42 tomato42 added this to the v0.14 milestone Nov 2, 2019
@tomato42 tomato42 self-assigned this Nov 2, 2019
@tomato42 tomato42 force-pushed the cosmic-ray branch 2 times, most recently from 3762ce6 to d1e39a5 Compare November 2, 2019 18:33
@coveralls
Copy link

coveralls commented Nov 2, 2019

Coverage Status

Coverage increased (+0.1%) to 98.699% when pulling 2273354 on tomato42:cosmic-ray into bbe3679 on warner:master.

@tomato42 tomato42 force-pushed the cosmic-ray branch 3 times, most recently from c51abb4 to 81dda36 Compare November 2, 2019 20:00
@tomato42 tomato42 modified the milestones: v0.14, someday/future Nov 3, 2019
@tomato42 tomato42 force-pushed the cosmic-ray branch 4 times, most recently from b3b5be8 to 8605a05 Compare December 3, 2019 01:37
@tomato42 tomato42 changed the title Add mutation testing to CI [WIP] Add mutation testing to CI Jan 3, 2020
@tomato42 tomato42 marked this pull request as draft August 25, 2020 16:10
@tomato42 tomato42 force-pushed the cosmic-ray branch 2 times, most recently from 1b1d3f9 to ca2079b Compare November 12, 2020 11:14
@tomato42
Copy link
Member Author

so, with those changes we end up with 222 test suites executed (6.7s per test suite execution), better, but not really good, that still means a 99.9% confidence interval of 10 percentage points

looks like we need to get down to around 1s per test suite execution to get workable confidence intervals

@tomato42
Copy link
Member Author

tomato42 commented Dec 6, 2020

@tomato42
Copy link
Member Author

tomato42 commented Dec 6, 2020

@tomato42
Copy link
Member Author

tomato42 commented Dec 7, 2020

@tomato42 tomato42 force-pushed the cosmic-ray branch 5 times, most recently from b8d3d3f to 967ed08 Compare January 4, 2024 01:01
@tomato42 tomato42 marked this pull request as ready for review January 15, 2024 01:27
@tomato42 tomato42 changed the title [WIP] Add mutation testing to CI Add mutation testing to CI Jan 15, 2024
Copy link

@The-Mule The-Mule left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only nitpicks. I did no spotted any obvious flaw in the changes.

systemd-run --user --scope -p MemoryMax=2G -p MemoryHigh=2G cosmic-ray --verbosity INFO exec cosmic-ray.toml session-vs-master.sqlite &
cosmic_pid=$!
for i in $(seq 1 600); do
# wait for test execution at most 10 minutes

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: Isn't it easier (if possible) to use timeout 10m?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea, but to implement it I would have to figure out error handling and so on... this is good enough (and probably easier to port to macos)

with self.assertRaises(ValueError) as e:
SigningKey.from_secret_exponent(1234567890, curve=Ed25519)

self.assertIn("don't support setting the secret", str(e.exception))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: exception message sounds a bit odd

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's because this isn't the whole message, the whole message is "Edwards keys don't support setting the secret scalar (exponent) directly"

except ImportError:
try:
from gmpy import mpz
except ImportError:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it safe the second ImportError to be completely silent?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why it wouldn't be? the point of this code is to define the mpz() method, where it comes from is secondary

@tomato42 tomato42 merged commit 0d5a38c into tlsfuzzer:master Jan 15, 2024
71 checks passed
@tomato42 tomato42 deleted the cosmic-ray branch January 15, 2024 21:23
@tomato42 tomato42 modified the milestones: someday/future, v0.19.0 Jan 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance issues related to making the project usable or testable
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants