executing cosmic-ray for subset of code?

[tomato42]

Please correct me if I'm wrong, but I couldn't find an example in documentation of integrating cosmic-ray with CI in a way that only changed code is tested.

While mutation testing is really nice, it is also slow. For verifying quality of test coverage added in PRs the execution needs to be relatively fast (tens of minutes not hours). But at the same time, most PRs change relatively small amount of code. Since small amount of code means fewer places to be mutated, that should lead to faster execution time.

So, is there a way to run cosmic-ray just to verify that the test coverage is correct and complete for a given patch, not the full code base?

[abingham]

There is currently no way to do that. It is, however, something we've long wanted to tackle.

I think our interceptor API gives us what we need to do this. We could make an interceptor that looks at a diff/patch and skips all mutations that don't touch code in the diff. Now that I've typed that, it actually sounds very doable, but I don't have time to put into right now. I'd be happy to help someone work on it, though.

[tomato42]

I was actually thinking of something much more crude and inefficient - collect all mutations and then delete ones from sqlite database that don't match lines changed in given patch :)

We could make an interceptor that looks at a diff/patch and skips all mutations that don't touch code in the diff.

what's the interface of an interceptor? an example of it?

I'd be happy to help someone work on it, though.

can't promise anything, but I've been itching for using mutation testing in my projects for quite some time...

[abingham]

I was actually thinking of something much more crude and inefficient - collect all mutations and then delete ones from sqlite database that don't match lines changed in given patch :)

That's a very good description of an interceptor! It's a plugin that runs after initialization. It's handed the fully populated WorkDB (i.e. the list of all the mutations to do) and it can manipulate it as necessary. The most common thing interceptors (currently) do is to find WorkItems matching some critera and mark them as "skipped".

An approachable example of an interceptor is cosmic_ray.interceptors.operators_filter.intercept. This looks through all of the WorkItems for those whose operators match a user-specified regular expressions, marking all matches as skipped. I think we could do something very similar for diffs: provide the diff via the config, look for matching lines, and skip all mutations that aren't in the diff.

[tomato42]

yeah, marking them as skipped is at least a little bit less crude

the up-to-date usage is described here?

[abingham]

Yes, that's incomplete but otherwise looks correct. What's really missing there is a description of how to write an interceptor. Currently you'd need to look at how an existing one is implemented, including how it's configured and how it's exposed to CR as a plugin. I think the "operators-filter" is a good example for that.