Extending metadata for "negative" tests

[jugglinmike]

Many tests in Test262 describe programs that are expected to fail. For the past year or so, we've been maintaining metadata to differentiate between two kinds of failures: the "phase" attribute. It's set to "early" for tests that should fail before the code is executed, and it's set to "runtime" for tests that should fail only after execution has begun. This allows JavaScript parsers to appropriately interpret test results: since parsers don't execute the code, they should expect that tests expected to fail at "runtime" do not fail to parse.

This distinction is a little muddy when it comes to module code. If one JavaScript module depends on another module and the second module has an "early" error, then running the first should produce that same early error. The problem is that it doesn't necessarily make sense for a JavaScript parser to load external modules. This means that a negative test marked as "early" might not actually make parsers fail. Currently, the folks running the code have to inspect these mismatches and make them as "allowed" on a case-by-case basis.

To make this experience better we can differentiate between these kinds of failure with a third value for the test metadata. That way, parsers could correctly interpret tests that are expected to fail when modules are being loaded (i.e. they should parse without an issue).

A few of us discussed this change at last week's TC39 meeting in Boston, and I'm excited to write the patch. Unfortunately, we stopped short of defining the specific "phase" metadata values we should use. I'd like to start the ball rolling:

• "parse" for all early errors that originate in the current module or the
  current script
• "resolution" for all early errors that originate in external module code
• "runtime" for all errors that occur during evaluation

I'm suggesting that we no longer use the value "early" because that word can describe both of the "phases" that we are trying to differentiate.

[adrianheine]

I think this can be closed now that #1366 is merged.

[rwaldron]

Like @adrianheine said, this is done.