Skip to content

Conversation

@pugnascotia
Copy link
Contributor

@pugnascotia pugnascotia commented Jul 22, 2022

We use between(x, y) calls with switch statements in tests to
randomize test behaviour. However, some usages define case statements
that can never execute, because the case value is outside the range
defined by the between call.

Write a rule that inspects the switches and the cases, and fails on the
broken cases.

The following issues cover the changes required to fix this problem.

We use `between(x, y)` calls with `switch` statements in tests to
randomize test behaviour. However, some usages define `case` statements
that can never execute, because the `case` value is outside the range
defined by the `between` call.

Write a rule that inspects the switches and the cases, and fails on the
broken cases.
@pugnascotia pugnascotia added >enhancement >test Issues or PRs that are addressing/adding tests :Delivery/Tooling Developer tooliing and automation v8.4.0 v7.17.6 labels Jul 22, 2022
@pugnascotia pugnascotia marked this pull request as draft July 22, 2022 16:43
@mark-vieira
Copy link
Contributor

@breskeby how hard would it be to add some benchmarks for this kind of thing or does the fact that our existing benchmark suite does precommit probably already account for this. The point being that complex checkstyle rules can be pretty expensive, so I want to ensure that we aren't adding to much overhead by adding rules for very specific edge case items like this.

@pugnascotia
Copy link
Contributor Author

@mark-vieira this implementation does at least limit itself to switch statements. It's not like the long-lines rules rule. But, I don't have any perf number for you, so 🤷‍♂️

@mark-vieira
Copy link
Contributor

@mark-vieira this implementation does at least limit itself to switch statements. It's not like the long-lines rules rule. But, I don't have any perf number for you, so 🤷‍♂️

Yeah, I don't suspect it'll be a problem, but we have infrastructure for generating performance metrics on the effect of any given build change so I think it's worth leveraging that for stuff like this (or any additional static analysis checks) to give a number to the cost of such a thing.

@elasticsearchmachine elasticsearchmachine changed the base branch from master to main July 22, 2022 23:04
@DaveCTurner
Copy link
Contributor

We also use switch (randomIntBetween(...)) and switch(randomInt(...)) which could suffer the same problems. Should we check these forms too? Or maybe we should drop these aliases and just use between everywhere?

@breskeby
Copy link
Contributor

@mark-vieira we do indirectly check the performance with the precommit checks we have. But given all the stuff we run there, I can imagine it would need to be really really significant for us to catch here. We could setup checkstyle specific performance checks to detect performance losses here early. I'm just not sure its worth it

@pugnascotia
Copy link
Contributor Author

pugnascotia commented Jul 25, 2022

@DaveCTurner I also handled randomIntBetween but I've added support for randomInt as well now.

@mark-vieira
Copy link
Contributor

We could setup checkstyle specific performance checks to detect performance losses here early.

I think it might be worth adding a scenario for checkstyle specifically at some point. We are increasingly adding more complex checkstyle rules and we don't really have any idea of how costly they are. Checkstyle as a whole we know is pretty expensive but it's not clear if that is just the AST generation or the rules, or what.

@pugnascotia
Copy link
Contributor Author

@elasticmachine update branch

@pugnascotia pugnascotia marked this pull request as ready for review July 26, 2022 13:32
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-delivery (Team:Delivery)

@elasticsearchmachine elasticsearchmachine added the Team:Delivery Meta label for Delivery team label Jul 26, 2022
@pugnascotia
Copy link
Contributor Author

@joegallo @mark-vieira the dependent issues are now closed, can I get a review please?

Copy link
Contributor

@joegallo joegallo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+100 on the net effect here, and the code looks reasonable to me. ❤️ for taking a look at this, @pugnascotia.

Copy link
Contributor

@mark-vieira mark-vieira left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pugnascotia pugnascotia merged commit 0df1438 into elastic:main Jul 27, 2022
@pugnascotia pugnascotia deleted the switch-between-checkstyle-rule branch July 27, 2022 08:22
pugnascotia added a commit that referenced this pull request Jul 27, 2022
We use `between(x, y)` calls with `switch` statements in tests to
randomize test behaviour. However, some usages define `case` statements
that can never execute, because the `case` value is outside the range
defined by the `between` call.

Write a rule that inspects the switches and the cases, and fails on the
broken cases. This rule checks `between`, `randomIntBetween` and
`randomInt`.
@pugnascotia
Copy link
Contributor Author

Backported to 7.17 in 50dc8ae (and fixed another error in 4c9f8b0).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Delivery/Tooling Developer tooliing and automation >enhancement Team:Delivery Meta label for Delivery team >test Issues or PRs that are addressing/adding tests v7.17.6 v8.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants