-
Notifications
You must be signed in to change notification settings - Fork 464
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add testing plan guide to documentation. #4113
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like a good first draft to me. Before it's ready for publicizing there are a few things I'd suggest:
- We should make the document more approachable with a couple of introductory paragraphs, explaining what it is and who it is intended for.
- We should have a section about diminishing returns and advice on where to draw the line. See, for example, the discussion at Math.sumPrecise testing plan #4054 and in Mike's testing plan guide for WPT.
- We should have a section on what steps to carry out, concretely. Things like, open a testing plan issue on the test262 repo. (Examples of good existing testing plan issues.) CC the proposal champions (if that is not you yourself). Send a PR to tc39/proposals to link the testing plan issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems like really good guidance. Any other maintainers want to take a look before merging?
|
||
#### What is this document | ||
|
||
[ECMAScript proposals](https://github.com/tc39/proposals/) that reach Stage 2.7 in the [TC39 Process](https://tc39.es/process-document/) need to have sufficient tests integrated to the Test262 test suite. To track the added tests, ensure that they are complete, and to get an ideao of the amount of work needed, it's often helpful to have a testing plan outlining what exactly should be tested. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[ECMAScript proposals](https://github.com/tc39/proposals/) that reach Stage 2.7 in the [TC39 Process](https://tc39.es/process-document/) need to have sufficient tests integrated to the Test262 test suite. To track the added tests, ensure that they are complete, and to get an ideao of the amount of work needed, it's often helpful to have a testing plan outlining what exactly should be tested. | |
[ECMAScript proposals](https://github.com/tc39/proposals/) that reach Stage 2.7 in the [TC39 Process](https://tc39.es/process-document/) need to have sufficient tests integrated to the Test262 test suite. To track the added tests, ensure that they are complete, and to get an idea of the amount of work needed, it's often helpful to have a testing plan outlining what exactly should be tested. |
|
||
#### Side-effects and algorithm step order | ||
|
||
Finally, consider testing the order of the algorithm steps, if there are steps that introduce side effects. However, it's best to not overdo it here, the goal is not to have 100% coverage, but to help implementations correctly implement the new functionality. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"the goal is not to have 100% coverage" — I'd make this more specific. We do want what most people understand as "100% coverage", but we don't want endless tests for every single permutation of user code calls, for example asserting the user calls for some ridiculous object like { toString() { return { toString() { return { toString() { ... } } } } } }
in a ToString algorithm step. Although this is already somewhat explained below, in "Completing the testing plan".
Maybe something like, "However, consider that exhaustively testing the order of side effects can lead to diminishing returns."
|
||
## Tips and tricks | ||
|
||
Check what engines test, in case you missed something. Some engine's test locations in their codebase: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Check what engines test, in case you missed something. Some engine's test locations in their codebase: | |
Check what engines test, in case you missed something. Some engines' test locations in their codebase: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want to link this from somewhere?
|
||
## Completing the testing plan | ||
|
||
As important as it is to have a complete testing plan, it is equally important to avoid diminishing returns and when to stop adding cases to be tested. Especially when new specification algorithms call on existing algorithms, it may be temping to dive in the existing algorithms to explore the different paths they may take. However it is best to focus on writing more surface tests for these cases, and only dive in the nested algorithms when the proposal is expecting these to be relevant to the new additions. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As important as it is to have a complete testing plan, it is equally important to avoid diminishing returns and when to stop adding cases to be tested. Especially when new specification algorithms call on existing algorithms, it may be temping to dive in the existing algorithms to explore the different paths they may take. However it is best to focus on writing more surface tests for these cases, and only dive in the nested algorithms when the proposal is expecting these to be relevant to the new additions. | |
As important as it is to have a complete testing plan, it is equally important to avoid diminishing returns and to understand when to stop adding cases to be tested. Especially when new specification algorithms call on existing algorithms, it may be temping to dive in the existing algorithms to explore the different paths they may take. However it is best to focus on writing more surface tests for these cases, and only dive in the nested algorithms when the proposal is expecting these to be relevant to the new additions. |
or something like it
A guide to help with writing testing plans for ECMAScript proposals in stage 2.7.