Skip to content

Conversation

@martincostello
Copy link
Member

Fix flaky mutants after task refactoring in #2664.

Fix flaky mutants after task refactoring in #2664.
Copilot AI review requested due to automatic review settings August 22, 2025 14:50
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR addresses flaky mutants in the Polly.Core circuit breaker controller by adding Stryker mutation testing exclusions. The changes disable mutation testing for specific conditional checks that are difficult to test reliably.

  • Added Stryker disable comments for two conditional checks
  • Prevents flaky mutation test failures for code paths that are hard to test deterministically

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@codecov
Copy link

codecov bot commented Aug 22, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.12%. Comparing base (0159145) to head (fc6b235).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2717   +/-   ##
=======================================
  Coverage   96.12%   96.12%           
=======================================
  Files         309      309           
  Lines        7118     7118           
  Branches     1008     1008           
=======================================
  Hits         6842     6842           
  Misses        222      222           
  Partials       54       54           
Flag Coverage Δ
linux 96.12% <ø> (ø)
macos 96.12% <ø> (ø)
windows 96.11% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pentp
Copy link
Contributor

pentp commented Aug 22, 2025

Isn't the Debug.Assert(task.IsCompleted) assert below those conditions enough to fix the mutation tests? IIRC mutation tests always run in debug mode and those asserts should ensure that the opposite condition will fail, no?
If that's not the case, then some of the other Debug.Asserts I added might also be flaky.

@martincostello
Copy link
Member Author

They are, but I've seen two different failing mutations since the merge.

"configuration": "Debug",

@pentp
Copy link
Contributor

pentp commented Aug 22, 2025

Oh, I think the reason is because Debug.Assert also gets mutated (removed) and if done together with the other mutation, then both will "survive". The reason I thought asserts would be enough is that in our codebase we have "ignore-methods": [ "Assert" ] in Stryker config and I had forgotten it's not the default.

@pentp
Copy link
Contributor

pentp commented Aug 22, 2025

This condition might also be flaky then:

if (!whenAnyHedgedTask.IsCompleted)

Though I'm surprised that the similar condition below for if (completedTask == delayTask) doesn't seem to be flaky...

@martincostello
Copy link
Member Author

I'd be happy to accept a PR to tweak our configuration to do the same and then revert these changes so the conditions are mutated again.

I'm going to merge this now though as 4-in-a-row failures in main and 3-in-a-row successes here suggests the flaky CI is at least resolved for now (🤞).

@martincostello martincostello merged commit 651508b into main Aug 22, 2025
40 checks passed
@martincostello martincostello deleted the fix-flaky-mutant branch August 22, 2025 15:19
@pentp
Copy link
Contributor

pentp commented Aug 22, 2025

It's actually in the config already, so now I don't know why it was flaky:

"Debug.Assert",

@martincostello
Copy link
Member Author

Just had another one happen here:

if (!whenAnyHedgedTask.IsCompleted)

LogicalNotExpression to un-LogicalNotExpression mutation Survived (132:13)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants