Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add testing plan guide to documentation. #4113

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

ioannad
Copy link
Contributor

@ioannad ioannad commented Jun 20, 2024

A guide to help with writing testing plans for ECMAScript proposals in stage 2.7.

@ioannad ioannad requested a review from a team as a code owner June 20, 2024 13:31
Copy link
Contributor

@ptomato ptomato left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a good first draft to me. Before it's ready for publicizing there are a few things I'd suggest:

  • We should make the document more approachable with a couple of introductory paragraphs, explaining what it is and who it is intended for.
  • We should have a section about diminishing returns and advice on where to draw the line. See, for example, the discussion at Math.sumPrecise testing plan #4054 and in Mike's testing plan guide for WPT.
  • We should have a section on what steps to carry out, concretely. Things like, open a testing plan issue on the test262 repo. (Examples of good existing testing plan issues.) CC the proposal champions (if that is not you yourself). Send a PR to tc39/proposals to link the testing plan issue.

docs/testing-plan-guide.md Outdated Show resolved Hide resolved
Copy link
Contributor

@ptomato ptomato left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like really good guidance. Any other maintainers want to take a look before merging?


#### What is this document

[ECMAScript proposals](https://github.com/tc39/proposals/) that reach Stage 2.7 in the [TC39 Process](https://tc39.es/process-document/) need to have sufficient tests integrated to the Test262 test suite. To track the added tests, ensure that they are complete, and to get an ideao of the amount of work needed, it's often helpful to have a testing plan outlining what exactly should be tested.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
[ECMAScript proposals](https://github.com/tc39/proposals/) that reach Stage 2.7 in the [TC39 Process](https://tc39.es/process-document/) need to have sufficient tests integrated to the Test262 test suite. To track the added tests, ensure that they are complete, and to get an ideao of the amount of work needed, it's often helpful to have a testing plan outlining what exactly should be tested.
[ECMAScript proposals](https://github.com/tc39/proposals/) that reach Stage 2.7 in the [TC39 Process](https://tc39.es/process-document/) need to have sufficient tests integrated to the Test262 test suite. To track the added tests, ensure that they are complete, and to get an idea of the amount of work needed, it's often helpful to have a testing plan outlining what exactly should be tested.


#### Side-effects and algorithm step order

Finally, consider testing the order of the algorithm steps, if there are steps that introduce side effects. However, it's best to not overdo it here, the goal is not to have 100% coverage, but to help implementations correctly implement the new functionality.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"the goal is not to have 100% coverage" — I'd make this more specific. We do want what most people understand as "100% coverage", but we don't want endless tests for every single permutation of user code calls, for example asserting the user calls for some ridiculous object like { toString() { return { toString() { return { toString() { ... } } } } } } in a ToString algorithm step. Although this is already somewhat explained below, in "Completing the testing plan".

Maybe something like, "However, consider that exhaustively testing the order of side effects can lead to diminishing returns."


## Tips and tricks

Check what engines test, in case you missed something. Some engine's test locations in their codebase:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Check what engines test, in case you missed something. Some engine's test locations in their codebase:
Check what engines test, in case you missed something. Some engines' test locations in their codebase:

Copy link
Contributor

@Ms2ger Ms2ger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to link this from somewhere?


## Completing the testing plan

As important as it is to have a complete testing plan, it is equally important to avoid diminishing returns and when to stop adding cases to be tested. Especially when new specification algorithms call on existing algorithms, it may be temping to dive in the existing algorithms to explore the different paths they may take. However it is best to focus on writing more surface tests for these cases, and only dive in the nested algorithms when the proposal is expecting these to be relevant to the new additions.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
As important as it is to have a complete testing plan, it is equally important to avoid diminishing returns and when to stop adding cases to be tested. Especially when new specification algorithms call on existing algorithms, it may be temping to dive in the existing algorithms to explore the different paths they may take. However it is best to focus on writing more surface tests for these cases, and only dive in the nested algorithms when the proposal is expecting these to be relevant to the new additions.
As important as it is to have a complete testing plan, it is equally important to avoid diminishing returns and to understand when to stop adding cases to be tested. Especially when new specification algorithms call on existing algorithms, it may be temping to dive in the existing algorithms to explore the different paths they may take. However it is best to focus on writing more surface tests for these cases, and only dive in the nested algorithms when the proposal is expecting these to be relevant to the new additions.

or something like it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants