Remove obsolete feature flags

[thejoshwolfe]

A number of feature flags are officially standard and are largely maintained for historical reasons in this test suite. I propose we remove some/all standard flags from features.txt and from the features: declarations from test scripts.

The value of old feature flags

Since "historical reasons" is pretty vague, let's discuss in more detail why old feature flags are valuable.

1. The flags already exist, and change is not free. It takes effort to remove the flags from test262, and it potentially takes effort for consumers to adapt to the change.
2. Consumers can filter which tests run while their implementation only supports a subset of the features. For example, someone starting a JavaScript implementation from scratch might not have arrow functions implemented yet, so they can skip all the tests that use arrow functions for the time being.

The negative value of old feature flags

The presence of the old feature flags is not free.

1. Dragging around a bunch of obsolete feature flags puts a burden on contributors writing new tests. For example, if you test that passing a Symbol for a particular parameter throws a TypeError, you need to remember to note the dependency on the Symbol feature. As the number of features increases, this burden increases.
2. It's not always clear how to describe your dependence on particular features when harness files get involved. See here. The fewer features exist, the smaller this problem becomes.

Accountability and precedent

There are many examples where tests do not properly declare their feature dependencies. Here is one way to find tests that neglect to declare their dependency on the Symbol feature:

test262$ grep -PL 'features: \[.*\bSymbol\b.*\]| - Symbol$' $(grep -RPl '\bSymbol\(' test/) | wc -l
117

I may just be oblivious, but I don't think this negligence is causing any problems for consumers. We might infer from this that omitting the Symbol feature dependency is harmless, which lowers the resistance to removing that feature throughout the codebase.

Git history is history

Depending on what "historical reasons" really means, we'll always have git history no matter what we do. If a consumer wants to try running this test suite and skip all the tests with a particular feature that has been removed, they might consider checking out an old git tag from before that feature was deleted.

Immediate proposal

It seems likely that at least some of the standard features are actually valuable to consumers despite being standardized. Even if Symbol isn't useful, it appears that Symbol.split might still be valuable to the Edge project, for example. Perhaps we could have a process for removing specific flags that we determine are no longer valuable, and in fact are negative value for the reasons mentioned.

A tool for editing feature dependency declarations would useful for here. As a start, it looks like we could delete the feature from features.txt and then use tools/lint/lib/checks/features.py to guide a human to edit the files manually.

Alternative

As an alternative to removing old features, could we perhaps look into enforcing more correctness in an automated way? My bash-grep-fu above is pretty rudimentary, but it's already better than nothing. Could we look into enforcing feature declarations for the easy cases, such as AST analysis to look for Symbol.toPrimitive and so forth?

[leobalter]

Dragging around a bunch of obsolete feature flags puts a burden on contributors writing new tests.

This project is consumed by many other projects. JS runtimes to start, but then you go to transpilers and several libraries. I was surprised this year when I knew of some projects still relying on ES6 features and how those are still necessary.

Calling features flags obsolete requires a minimal research on who is consuming this. If we base in a single consumer project, this would be just a subjective and arbitrary decision.

The complexity of a project is not a burden when it's necessary for proper maintenance.

I propose we remove some/all standard flags from features.txt

all flags? This is again another subjective and arbitrary suggestion. I wish the log could have shown a better reason why we just documented the features flags in a file. This is raising back a discussion we recently had a conclusion.

It's not always clear how to describe your dependence on particular features when harness files get involved.

This is because harness files should be universal helpers with direct purposes. That's the same reason we have specialized helpers like the ones for TypedArrays, they rely on the respective constructors, but not anything else that could come as a surprise. E.g.: async iterators.

There are many examples where tests do not properly declare their feature dependencies.

This is an issue to be solved, but not to kill a useful pattern. You already proposed something to start, and we also have plans to solve this as well.

Unfortunately, time is limited and there are so many priorities I have to deal first, that doesn't mean I'll sacrifice something that is not hurting while it is not fully complete. This project has met a lot of significant improvements in the last two years and I don't think we should stop doing it at this point.

If a consumer wants to try running this test suite and skip all the tests with a particular feature that has been removed, they might consider checking out an old git tag from before that feature was deleted.

This is biased. I once believed this could be an option, but if we don't care about releasing tagged versions for specific features, we can't really say someone else could just checkout onto anything.

This is a live project, as ECMAScript is. There are many fixes for tests, fixes for the specs themselves, changes in proposals, some of them reflecting web reality. There's no versioning point to correspond to some specific feature. Everything is live, organic, and we can't be rude to other consumers just to make a biased decision for a few of us.

As an alternative to removing old features, could we perhaps look into enforcing more correctness in an automated way? My bash-grep-fu above is pretty rudimentary, but it's already better than nothing. Could we look into enforcing feature declarations for the easy cases, such as AST analysis to look for Symbol.toPrimitive and so forth?

I wish this issue could be just about this. This is something we have as a plan, as this is a perceivable improvement that won't hurt anyone, but enhance the quality of Test262.

As I mentioned, we need to find availability for improvements and that includes prioritization. Writing new tests for Stage 3+ proposals and normative PRs and bug triaging are on top of this prioritization list.

Test262 is not a racing game.

[rwaldron]

w/r to:

Could we look into enforcing feature declarations for the easy cases, such as AST analysis to look for Symbol.toPrimitive and so forth?

I'd like to add to @leobalter's response and point out that this would also require maintaining a parser that supported Stage 3 (maybe even Stage 2) features. While we'd like to offer resources to Babylon to do this, we can't make that a blocker when simply typing the features in all new test files is 100% sufficient.

[thejoshwolfe]

This is because harness files should be universal helpers with direct purposes. That's the same reason we have specialized helpers like the ones for TypedArrays, they rely on the respective constructors, but not anything else that could come as a surprise. E.g.: async iterators.

#1200 doesn't fit this philosophy. There, a harness file wants to test any strange primitive types that exist, like Symbol and BigInt. But I suppose this is a bit off topic from the original issue here.

Calling features flags obsolete requires a minimal research on who is consuming this. If we base in a single consumer project, this would be just a subjective and arbitrary decision.

My research was that the Symbol feature is not properly referenced here, and no one has filed a bug against this project mentioning that. That means that consumers either haven't noticed or haven't demonstrated that they care. Is that not evidence that the Symbol feature can be removed?

[rwaldron]

My research was that the Symbol feature is not properly referenced here,

Sorry, can clarify where "here" is?

[thejoshwolfe]

Sorry, can clarify where "here" is?

test262$ grep -PL 'features: \[.*\bSymbol\b.*\]| - Symbol$' $(grep -RPl '\bSymbol\(' test/) | head
test/built-ins/WeakSet/symbol-disallowed-as-weakset-key.js
test/built-ins/WeakSet/prototype/delete/this-not-object-throw-symbol.js
test/built-ins/WeakMap/prototype/get/returns-undefined-key-is-not-object.js
test/built-ins/Map/prototype/size/this-not-object-throw.js
test/built-ins/TypedArrays/buffer-arg-byteoffset-is-symbol-throws-sab.js
test/built-ins/TypedArrays/buffer-arg-length-is-symbol-throws-sab.js
test/built-ins/Boolean/symbol-coercion.js
test/built-ins/Set/prototype/delete/this-not-object-throw-symbol.js
test/built-ins/Set/prototype/entries/this-not-object-throw-symbol.js
test/built-ins/Set/prototype/add/this-not-object-throw-symbol.js

[leobalter]

btw, this command will allow using git grep without requiring libpcre.

git grep  -LE 'features: \[.*\bSymbol\b.*\]| - Symbol$' $(git grep --name-only -E '\bSymbol\(' test/)

[rwaldron]

Sorry, can clarify where "here" is?
test262$ grep -PL 'features: [.\bSymbol\b.]| - Symbol$' $(grep -RPl '\bSymbol(' test/)
...

Thanks for that. I understand that you're saying the features front matter in Test262 is presently incomplete, and specifically calling out Symbol. This is absolutely true, but not intentional. Our goal is to eventually include features front matter in every test file, to enable fine grained categorization and targeted test running. This must be implemented as static meta data to account for all of the Test262 consumer use cases that exist, as not all engine/runtime authors are using the test material in the same or intended way. Our biggest "feature" request, above all others, is expansion of the features front matter—that means Test262 is going to have more and finer grained features coverage. Taking into consideration the existing expectations and the present issue that #1200 faces, @jugglinmike, @leobalter and I have come up with a solution that will meet all of our needs:

1. Creation of an index of required features flags that are associated with each harness/* file. The index will be documented in metadata
2. The index described above will be enforced by extensions to the linter

This strategy will unblock #1200

[thejoshwolfe]

a solution that will meet all of our needs

Great news! Is there somewhere I can get more details on this? Anything I can do to help make it happen?

For #1200, will I be able to conditionally test BigInt if it exists, or will the harness file declare an unconditional dependency on BigInt, thereby causing the String.prototype.indexOf parameter type coercion test to be entirely skipped when BigInt is not supported? Or will the harness file just have to do without testing BigInt parameters?

[rwaldron]

Is there somewhere I can get more details on this? Anything I can do to help make it happen?

I'm writing the linting rules now, using your patch as the base branch

will I be able to conditionally test BigInt if it exists

Yes, you can absolutely do that!

or will the harness file declare an unconditional dependency on BigInt,

The features flags are "required", but they aren't unconditional dependencies. BigInt will become a features flag that will be included in eg. test/built-ins/String/prototype/indexOf/position-tointeger.js, but that test will always run by default—features flags are to be used for "opt in" filtering.

In summary, the only change that will be necessary (and which I will make in my new commits) is the addition of features: [Symbol.toPrimitive,BigInt] to both test/built-ins/String/prototype/indexOf/position-tointeger.js and test/built-ins/String/prototype/indexOf/searchstring-tostring.js

[anba]

FYI: List of tests which probably use some kind of Symbol functionality, but not declaring any of the Symbol features tags:

test\built-ins\Array\from\source-object-iterator-1.js
test\built-ins\Array\from\source-object-iterator-2.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_array-like-length-to-string-throws.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_array-like-length-value-of-throws.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_array-like-negative-length.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_array-like-primitive-non-number-length.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_array-like-string-length.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_array-like-to-length-throws.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_array-like.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_holey-sloppy-arguments.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_large-typed-array.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_length-throws.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_sloppy-arguments-throws.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_sloppy-arguments-with-dupes.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_sloppy-arguments.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_small-typed-array.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_spreadable-boolean-wrapper.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_spreadable-function.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_spreadable-getter-throws.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_spreadable-number-wrapper.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_spreadable-reg-exp.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_spreadable-sparse-object.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_spreadable-string-wrapper.js
test\built-ins\Array\prototype\concat\Array.prototype.concat_strict-arguments.js
test\built-ins\Array\prototype\concat\create-proto-from-ctor-realm-array.js
test\built-ins\Array\prototype\concat\create-proto-from-ctor-realm-non-array.js
test\built-ins\Array\prototype\filter\create-proto-from-ctor-realm-array.js
test\built-ins\Array\prototype\filter\create-proto-from-ctor-realm-non-array.js
test\built-ins\Array\prototype\map\create-proto-from-ctor-realm-array.js
test\built-ins\Array\prototype\map\create-proto-from-ctor-realm-non-array.js
test\built-ins\Array\prototype\slice\create-proto-from-ctor-realm-array.js
test\built-ins\Array\prototype\slice\create-proto-from-ctor-realm-non-array.js
test\built-ins\Array\prototype\splice\create-proto-from-ctor-realm-array.js
test\built-ins\Array\prototype\splice\create-proto-from-ctor-realm-non-array.js
test\built-ins\Array\prototype\splice\create-species-length-exceeding-integer-limit.js
test\built-ins\ArrayIteratorPrototype\next\args-mapped-expansion-after-exhaustion.js
test\built-ins\ArrayIteratorPrototype\next\args-mapped-expansion-before-exhaustion.js
test\built-ins\ArrayIteratorPrototype\next\args-mapped-iteration.js
test\built-ins\ArrayIteratorPrototype\next\args-mapped-truncation-before-exhaustion.js
test\built-ins\ArrayIteratorPrototype\next\args-unmapped-expansion-after-exhaustion.js
test\built-ins\ArrayIteratorPrototype\next\args-unmapped-expansion-before-exhaustion.js
test\built-ins\ArrayIteratorPrototype\next\args-unmapped-iteration.js
test\built-ins\ArrayIteratorPrototype\next\args-unmapped-truncation-before-exhaustion.js
test\built-ins\ArrayIteratorPrototype\next\Float32Array.js
test\built-ins\ArrayIteratorPrototype\next\Float64Array.js
test\built-ins\ArrayIteratorPrototype\next\Int16Array.js
test\built-ins\ArrayIteratorPrototype\next\Int32Array.js
test\built-ins\ArrayIteratorPrototype\next\Int8Array.js
test\built-ins\ArrayIteratorPrototype\next\iteration-mutable.js
test\built-ins\ArrayIteratorPrototype\next\iteration.js
test\built-ins\ArrayIteratorPrototype\next\length.js
test\built-ins\ArrayIteratorPrototype\next\name.js
test\built-ins\ArrayIteratorPrototype\next\non-own-slots.js
test\built-ins\ArrayIteratorPrototype\next\property-descriptor.js
test\built-ins\ArrayIteratorPrototype\next\Uint16Array.js
test\built-ins\ArrayIteratorPrototype\next\Uint32Array.js
test\built-ins\ArrayIteratorPrototype\next\Uint8Array.js
test\built-ins\ArrayIteratorPrototype\next\Uint8ClampedArray.js
test\built-ins\ArrayIteratorPrototype\Symbol.toStringTag\value-direct.js
test\built-ins\ArrayIteratorPrototype\Symbol.toStringTag\value-from-to-string.js
test\built-ins\AsyncFunction\AsyncFunctionPrototype-to-string.js
test\built-ins\Atomics\add\non-views.js
test\built-ins\Atomics\and\non-views.js
test\built-ins\Atomics\compareExchange\non-views.js
test\built-ins\Atomics\exchange\non-views.js
test\built-ins\Atomics\load\non-views.js
test\built-ins\Atomics\or\non-views.js
test\built-ins\Atomics\store\non-views.js
test\built-ins\Atomics\sub\non-views.js
test\built-ins\Atomics\wait\non-views.js
test\built-ins\Atomics\wake\non-views.js
test\built-ins\Atomics\xor\non-views.js
test\built-ins\Function\prototype\toString\intrinsics.js
test\built-ins\Object\prototype\toString\proxy-function.js
test\built-ins\Promise\prototype\then\ctor-throws.js
test\built-ins\RegExp\prototype\Symbol.match\length.js
test\built-ins\RegExp\prototype\Symbol.match\name.js
test\built-ins\RegExp\prototype\Symbol.match\prop-desc.js
test\built-ins\RegExp\prototype\Symbol.replace\length.js
test\built-ins\RegExp\prototype\Symbol.replace\name.js
test\built-ins\RegExp\prototype\Symbol.replace\prop-desc.js
test\built-ins\RegExp\prototype\Symbol.split\length.js
test\built-ins\RegExp\prototype\Symbol.split\name.js
test\built-ins\Set\prototype\Symbol.toStringTag\property-descriptor.js
test\built-ins\Set\set-iterator-close-after-add-failure.js
test\built-ins\Set\set-iterator-next-failure.js
test\built-ins\Set\set-iterator-value-failure.js
test\built-ins\StringIteratorPrototype\ancestry.js
test\built-ins\StringIteratorPrototype\next\name.js
test\built-ins\StringIteratorPrototype\next\next-iteration-surrogate-pairs.js
test\built-ins\StringIteratorPrototype\next\next-iteration.js
test\built-ins\StringIteratorPrototype\next\next-missing-internal-slots.js
test\built-ins\Symbol\for\cross-realm.js
test\built-ins\Symbol\for\length.js
test\built-ins\Symbol\for\name.js
test\built-ins\Symbol\for\prop-desc.js
test\built-ins\Symbol\for\retrieve-value.js
test\built-ins\Symbol\keyFor\arg-symbol-registry-hit.js
test\built-ins\Symbol\keyFor\cross-realm.js
test\built-ins\Symbol\keyFor\length.js
test\built-ins\Symbol\keyFor\name.js
test\built-ins\Symbol\keyFor\prop-desc.js
test\built-ins\Symbol\length.js
test\built-ins\Symbol\name.js
test\built-ins\Symbol\prototype\constructor.js
test\built-ins\Symbol\prototype\toString\length.js
test\built-ins\Symbol\prototype\toString\name.js
test\built-ins\Symbol\prototype\toString\prop-desc.js
test\built-ins\Symbol\prototype\valueOf\length.js
test\built-ins\Symbol\prototype\valueOf\name.js
test\built-ins\Symbol\prototype\valueOf\prop-desc.js
test\built-ins\Symbol\prototype\valueOf\this-val-non-obj.js
test\built-ins\Symbol\prototype\valueOf\this-val-obj-non-symbol.js
test\built-ins\Symbol\species\basic.js
test\built-ins\Symbol\species\builtin-getter-name.js
test\built-ins\Symbol\species\subclassing.js
test\built-ins\Symbol\symbol.js
test\built-ins\TypedArray\prototype\Symbol.toStringTag\detached-buffer.js
test\built-ins\TypedArray\prototype\Symbol.toStringTag\length.js
test\built-ins\WeakMap\iterator-close-after-set-failure.js
test\built-ins\WeakMap\iterator-next-failure.js
test\built-ins\WeakMap\iterator-value-failure.js
test\built-ins\WeakSet\iterator-close-after-add-failure.js
test\built-ins\WeakSet\iterator-next-failure.js
test\built-ins\WeakSet\iterator-value-failure.js
test\language\expressions\arrow-function\dstr-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\arrow-function\dstr-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\arrow-function\dstr-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\arrow-function\dstr-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\async-generator\dstr-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\async-generator\dstr-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\async-generator\dstr-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\async-generator\dstr-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\async-generator\dstr-named-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\async-generator\dstr-named-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\async-generator\dstr-named-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\async-generator\dstr-named-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\async-generator\named-yield-star-getiter-sync-not-callable-boolean-throw.js
test\language\expressions\async-generator\named-yield-star-getiter-sync-not-callable-number-throw.js
test\language\expressions\async-generator\named-yield-star-getiter-sync-not-callable-object-throw.js
test\language\expressions\async-generator\named-yield-star-getiter-sync-not-callable-string-throw.js
test\language\expressions\async-generator\named-yield-star-getiter-sync-returns-boolean-throw.js
test\language\expressions\async-generator\named-yield-star-getiter-sync-returns-null-throw.js
test\language\expressions\async-generator\named-yield-star-getiter-sync-returns-number-throw.js
test\language\expressions\async-generator\named-yield-star-getiter-sync-returns-string-throw.js
test\language\expressions\async-generator\named-yield-star-getiter-sync-returns-symbol-throw.js
test\language\expressions\async-generator\named-yield-star-getiter-sync-returns-undefined-throw.js
test\language\expressions\async-generator\named-yield-star-sync-throw.js
test\language\expressions\async-generator\yield-star-getiter-sync-not-callable-boolean-throw.js
test\language\expressions\async-generator\yield-star-getiter-sync-not-callable-number-throw.js
test\language\expressions\async-generator\yield-star-getiter-sync-not-callable-object-throw.js
test\language\expressions\async-generator\yield-star-getiter-sync-not-callable-string-throw.js
test\language\expressions\async-generator\yield-star-getiter-sync-returns-boolean-throw.js
test\language\expressions\async-generator\yield-star-getiter-sync-returns-null-throw.js
test\language\expressions\async-generator\yield-star-getiter-sync-returns-number-throw.js
test\language\expressions\async-generator\yield-star-getiter-sync-returns-string-throw.js
test\language\expressions\async-generator\yield-star-getiter-sync-returns-symbol-throw.js
test\language\expressions\async-generator\yield-star-getiter-sync-returns-undefined-throw.js
test\language\expressions\async-generator\yield-star-sync-throw.js
test\language\expressions\class\async-gen-method-static-yield-star-getiter-sync-not-callable-boolean-throw.js
test\language\expressions\class\async-gen-method-static-yield-star-getiter-sync-not-callable-number-throw.js
test\language\expressions\class\async-gen-method-static-yield-star-getiter-sync-not-callable-object-throw.js
test\language\expressions\class\async-gen-method-static-yield-star-getiter-sync-not-callable-string-throw.js
test\language\expressions\class\async-gen-method-static-yield-star-getiter-sync-returns-boolean-throw.js
test\language\expressions\class\async-gen-method-static-yield-star-getiter-sync-returns-null-throw.js
test\language\expressions\class\async-gen-method-static-yield-star-getiter-sync-returns-number-throw.js
test\language\expressions\class\async-gen-method-static-yield-star-getiter-sync-returns-string-throw.js
test\language\expressions\class\async-gen-method-static-yield-star-getiter-sync-returns-symbol-throw.js
test\language\expressions\class\async-gen-method-static-yield-star-getiter-sync-returns-undefined-throw.js
test\language\expressions\class\async-gen-method-static-yield-star-sync-throw.js
test\language\expressions\class\async-gen-method-yield-star-getiter-sync-not-callable-boolean-throw.js
test\language\expressions\class\async-gen-method-yield-star-getiter-sync-not-callable-number-throw.js
test\language\expressions\class\async-gen-method-yield-star-getiter-sync-not-callable-object-throw.js
test\language\expressions\class\async-gen-method-yield-star-getiter-sync-not-callable-string-throw.js
test\language\expressions\class\async-gen-method-yield-star-getiter-sync-returns-boolean-throw.js
test\language\expressions\class\async-gen-method-yield-star-getiter-sync-returns-null-throw.js
test\language\expressions\class\async-gen-method-yield-star-getiter-sync-returns-number-throw.js
test\language\expressions\class\async-gen-method-yield-star-getiter-sync-returns-string-throw.js
test\language\expressions\class\async-gen-method-yield-star-getiter-sync-returns-symbol-throw.js
test\language\expressions\class\async-gen-method-yield-star-getiter-sync-returns-undefined-throw.js
test\language\expressions\class\async-gen-method-yield-star-sync-throw.js
test\language\expressions\class\dstr-async-gen-meth-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\class\dstr-async-gen-meth-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\class\dstr-async-gen-meth-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\class\dstr-async-gen-meth-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\class\dstr-async-gen-meth-static-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\class\dstr-async-gen-meth-static-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\class\dstr-async-gen-meth-static-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\class\dstr-async-gen-meth-static-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\class\dstr-gen-meth-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\class\dstr-gen-meth-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\class\dstr-gen-meth-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\class\dstr-gen-meth-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\class\dstr-gen-meth-static-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\class\dstr-gen-meth-static-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\class\dstr-gen-meth-static-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\class\dstr-gen-meth-static-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\class\dstr-meth-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\class\dstr-meth-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\class\dstr-meth-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\class\dstr-meth-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\class\dstr-meth-static-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\class\dstr-meth-static-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\class\dstr-meth-static-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\class\dstr-meth-static-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\function\dstr-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\function\dstr-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\function\dstr-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\function\dstr-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\generators\dstr-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\generators\dstr-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\generators\dstr-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\generators\dstr-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\object\dstr-async-gen-meth-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\object\dstr-async-gen-meth-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\object\dstr-async-gen-meth-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\object\dstr-async-gen-meth-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\object\dstr-gen-meth-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\object\dstr-gen-meth-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\object\dstr-gen-meth-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\object\dstr-gen-meth-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\object\dstr-meth-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\object\dstr-meth-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\object\dstr-meth-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\expressions\object\dstr-meth-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\expressions\object\method-definition\async-gen-yield-star-getiter-sync-not-callable-boolean-throw.js
test\language\expressions\object\method-definition\async-gen-yield-star-getiter-sync-not-callable-number-throw.js
test\language\expressions\object\method-definition\async-gen-yield-star-getiter-sync-not-callable-object-throw.js
test\language\expressions\object\method-definition\async-gen-yield-star-getiter-sync-not-callable-string-throw.js
test\language\expressions\object\method-definition\async-gen-yield-star-getiter-sync-returns-boolean-throw.js
test\language\expressions\object\method-definition\async-gen-yield-star-getiter-sync-returns-null-throw.js
test\language\expressions\object\method-definition\async-gen-yield-star-getiter-sync-returns-number-throw.js
test\language\expressions\object\method-definition\async-gen-yield-star-getiter-sync-returns-string-throw.js
test\language\expressions\object\method-definition\async-gen-yield-star-getiter-sync-returns-symbol-throw.js
test\language\expressions\object\method-definition\async-gen-yield-star-getiter-sync-returns-undefined-throw.js
test\language\expressions\object\method-definition\async-gen-yield-star-sync-throw.js
test\language\expressions\yield\star-iterable.js
test\language\statements\async-generator\dstr-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\async-generator\dstr-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\async-generator\dstr-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\async-generator\dstr-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\async-generator\yield-star-getiter-sync-not-callable-boolean-throw.js
test\language\statements\async-generator\yield-star-getiter-sync-not-callable-number-throw.js
test\language\statements\async-generator\yield-star-getiter-sync-not-callable-object-throw.js
test\language\statements\async-generator\yield-star-getiter-sync-not-callable-string-throw.js
test\language\statements\async-generator\yield-star-getiter-sync-returns-boolean-throw.js
test\language\statements\async-generator\yield-star-getiter-sync-returns-null-throw.js
test\language\statements\async-generator\yield-star-getiter-sync-returns-number-throw.js
test\language\statements\async-generator\yield-star-getiter-sync-returns-string-throw.js
test\language\statements\async-generator\yield-star-getiter-sync-returns-symbol-throw.js
test\language\statements\async-generator\yield-star-getiter-sync-returns-undefined-throw.js
test\language\statements\async-generator\yield-star-sync-throw.js
test\language\statements\class\async-gen-method-static-yield-star-getiter-sync-not-callable-boolean-throw.js
test\language\statements\class\async-gen-method-static-yield-star-getiter-sync-not-callable-number-throw.js
test\language\statements\class\async-gen-method-static-yield-star-getiter-sync-not-callable-object-throw.js
test\language\statements\class\async-gen-method-static-yield-star-getiter-sync-not-callable-string-throw.js
test\language\statements\class\async-gen-method-static-yield-star-getiter-sync-returns-boolean-throw.js
test\language\statements\class\async-gen-method-static-yield-star-getiter-sync-returns-null-throw.js
test\language\statements\class\async-gen-method-static-yield-star-getiter-sync-returns-number-throw.js
test\language\statements\class\async-gen-method-static-yield-star-getiter-sync-returns-string-throw.js
test\language\statements\class\async-gen-method-static-yield-star-getiter-sync-returns-symbol-throw.js
test\language\statements\class\async-gen-method-static-yield-star-getiter-sync-returns-undefined-throw.js
test\language\statements\class\async-gen-method-static-yield-star-sync-throw.js
test\language\statements\class\async-gen-method-yield-star-getiter-sync-not-callable-boolean-throw.js
test\language\statements\class\async-gen-method-yield-star-getiter-sync-not-callable-number-throw.js
test\language\statements\class\async-gen-method-yield-star-getiter-sync-not-callable-object-throw.js
test\language\statements\class\async-gen-method-yield-star-getiter-sync-not-callable-string-throw.js
test\language\statements\class\async-gen-method-yield-star-getiter-sync-returns-boolean-throw.js
test\language\statements\class\async-gen-method-yield-star-getiter-sync-returns-null-throw.js
test\language\statements\class\async-gen-method-yield-star-getiter-sync-returns-number-throw.js
test\language\statements\class\async-gen-method-yield-star-getiter-sync-returns-string-throw.js
test\language\statements\class\async-gen-method-yield-star-getiter-sync-returns-symbol-throw.js
test\language\statements\class\async-gen-method-yield-star-getiter-sync-returns-undefined-throw.js
test\language\statements\class\async-gen-method-yield-star-sync-throw.js
test\language\statements\class\dstr-async-gen-meth-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\class\dstr-async-gen-meth-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\class\dstr-async-gen-meth-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\class\dstr-async-gen-meth-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\class\dstr-async-gen-meth-static-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\class\dstr-async-gen-meth-static-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\class\dstr-async-gen-meth-static-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\class\dstr-async-gen-meth-static-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\class\dstr-gen-meth-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\class\dstr-gen-meth-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\class\dstr-gen-meth-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\class\dstr-gen-meth-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\class\dstr-gen-meth-static-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\class\dstr-gen-meth-static-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\class\dstr-gen-meth-static-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\class\dstr-gen-meth-static-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\class\dstr-meth-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\class\dstr-meth-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\class\dstr-meth-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\class\dstr-meth-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\class\dstr-meth-static-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\class\dstr-meth-static-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\class\dstr-meth-static-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\class\dstr-meth-static-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\class\subclass\builtin-objects\Symbol\new-symbol-with-super-throws.js
test\language\statements\class\subclass\builtin-objects\Symbol\symbol-valid-as-extends-value.js
test\language\statements\const\dstr-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\const\dstr-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\for\dstr-const-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\for\dstr-const-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\for\dstr-let-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\for\dstr-let-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\for\dstr-var-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\for\dstr-var-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\for-await-of\async-func-dstr-const-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\for-await-of\async-func-dstr-const-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\for-await-of\async-func-dstr-let-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\for-await-of\async-func-dstr-let-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\for-await-of\async-func-dstr-var-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\for-await-of\async-func-dstr-var-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\for-await-of\async-gen-dstr-const-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\for-await-of\async-gen-dstr-const-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\for-await-of\async-gen-dstr-let-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\for-await-of\async-gen-dstr-let-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\for-await-of\async-gen-dstr-var-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\for-await-of\async-gen-dstr-var-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\for-of\Array.prototype.Symbol.iterator.js
test\language\statements\for-of\dstr-const-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\for-of\dstr-const-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\for-of\dstr-let-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\for-of\dstr-let-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\for-of\dstr-var-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\for-of\dstr-var-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\function\dstr-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\function\dstr-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\function\dstr-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\function\dstr-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\generators\dstr-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\generators\dstr-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\generators\dstr-dflt-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\generators\dstr-dflt-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\let\dstr-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\let\dstr-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\try\dstr-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\try\dstr-ary-ptrn-elem-id-iter-val-err.js
test\language\statements\variable\dstr-ary-ptrn-elem-id-iter-step-err.js
test\language\statements\variable\dstr-ary-ptrn-elem-id-iter-val-err.js

Generated by excluding all tests which contain any of "Symbol,Symbol.hasInstance,Symbol.isConcatSpreadable,Symbol.iterator,Symbol.match,Symbol.replace,Symbol.search,Symbol.species,Symbol.split,Symbol.toPrimitive,Symbol.toStringTag,Symbol.unscopables" and then removing the global Symbol property.
Edited: Forgot to exclude Symbol.asyncIterator.

[rwaldron]

@anba lol... thanks :P this is all part of our current workload to get fixed

[anba]

@rwaldron Oh no, I've forgot to exclude Symbol.asyncIterator! That means there are only ~340 instead of ~700 files to update. 😄

[rwaldron]

😂🤣

[leobalter]

Thanks, @anba! btw, are you using Windows??

[anba]

@leobalter From time to time. 😄