-
Notifications
You must be signed in to change notification settings - Fork 464
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add testing plan guide to documentation. #4113
Open
ioannad
wants to merge
3
commits into
tc39:main
Choose a base branch
from
ioannad:testing-plan-guide
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 1 commit
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,69 @@ | ||||||
# Testing plan guide | ||||||
|
||||||
|
||||||
Understand the proposal text: | ||||||
|
||||||
- read the explainer in the proposal repo's README, and | ||||||
- read the slides from any TC39 plenary presentation. | ||||||
|
||||||
If you're unfamiliar with ECMAScript spec text, read this guide on [how to read the ECMAScript Specification](https://timothygu.me/es-howto/). | ||||||
|
||||||
Read the proposal's rendered specification. | ||||||
|
||||||
## For new APIs | ||||||
|
||||||
### Boilerplate tests | ||||||
|
||||||
For new APIs, include standard tests: | ||||||
|
||||||
- Property descriptor of the object/method/whatever. [Example](https://github.com/tc39/test262/blob/main/test/built-ins/String/prototype/at/prop-desc.js) | ||||||
- Brand check for methods, what happens when calling it with a `this` that is not what what it expects, via `Function.prototype.apply` or `.call`. [Example](https://github.com/tc39/test262/blob/main/test/intl402/ListFormat/prototype/format/branding.js) | ||||||
- Object prototype and extensibility. [Example](https://github.com/tc39/test262/blob/main/test/built-ins/Temporal/Instant/builtin.js) | ||||||
- Function boilerplate: | ||||||
+ Function length. [Example](https://github.com/tc39/test262/blob/main/test/built-ins/String/prototype/at/length.js) | ||||||
+ Function name. [Example](https://github.com/tc39/test262/blob/main/test/built-ins/String/prototype/at/name.js) | ||||||
+ Function is or is not a constructor. [Example](https://github.com/tc39/test262/blob/main/test/built-ins/Number/isNaN/not-a-constructor.js) | ||||||
|
||||||
### Success cases | ||||||
|
||||||
List the various success cases that result in a normal completion: | ||||||
|
||||||
- Identify algorithm branches, and include a potential input that leads to each branch case. | ||||||
- See the proposal's explainer for particular cases that this proposal wants to cover. | ||||||
|
||||||
### Failure cases | ||||||
|
||||||
List failure cases, when an exception is thrown: | ||||||
|
||||||
- Check new spec text for every place where a `?` appears, indicating that what follows may throw an exception. | ||||||
- List other potential unexpected edge cases. At the rendered spec, use the `U` key to see who calls the related parts of the code, and check for unexpected entry points. | ||||||
|
||||||
### Side-effects and algorithm step order | ||||||
|
||||||
Finally, consider testing the order of the algorithm steps, if there are steps that introduce side effects. However, it's best to not overdo it here, the goal is not to have 100% coverage, but to help implementations correctly implement the new functionality. | ||||||
|
||||||
## For new syntax | ||||||
|
||||||
A testing plan for new syntax is similar to that for a new API but additionally: | ||||||
|
||||||
- Test both the syntax and how it's evaluated, so double the tests. | ||||||
- Check [CONTRIBUTING.md](https://github.com/tc39/test262/blob/main/CONTRIBUTING.md) for special frontmatter flags when testing syntax errors. | ||||||
- Look into the ECMA-262 sections named [Runtime Semantics: Evaluation](https://tc39.es/ecma262/#sec-array-initializer-runtime-semantics-evaluation). | ||||||
|
||||||
## Tips and tricks | ||||||
|
||||||
Check what engines test, in case you missed something. Some engine's test locations in their codebase: | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
- v8 `v8/test/mjsunit/` | ||||||
- SpiderMonkey: `js/src/tests/non262/` | ||||||
ptomato marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
- JSC `JSTests/` | ||||||
|
||||||
Finally, if you don't understand why a certain part of the spec is the way it is, it helps to git blame the file `spec.html` of the proposal git repository. You may find linked PRs pointing to related issues, potentially with test cases that cover that part of the spec. | ||||||
|
||||||
### Examples of input categories | ||||||
|
||||||
- `undefined`, `null`, symbols | ||||||
- strings, "", "\b", "\n" | ||||||
- Booleans | ||||||
- number, BigInt, NaN, ±0, ±∞ | ||||||
- Objects, Arrays |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"the goal is not to have 100% coverage" — I'd make this more specific. We do want what most people understand as "100% coverage", but we don't want endless tests for every single permutation of user code calls, for example asserting the user calls for some ridiculous object like
{ toString() { return { toString() { return { toString() { ... } } } } } }
in a ToString algorithm step. Although this is already somewhat explained below, in "Completing the testing plan".Maybe something like, "However, consider that exhaustively testing the order of side effects can lead to diminishing returns."