Use a deterministic depth-first order when managing dependencies

[ChadKillingsworth]

Warning: this is potentially a high-impact change.

Due to the history of ordering dependencies as scripts with goog.provide and goog.require calls, the dependency sorting could produce incorrect results when using ES6 or CommonJS modules. In addition, the regex based parsing of dependencies did not recognize destructuring imports that spanned more than one line - and these are quite common external to Google.

This change begins with entry points and fully parses each dependency as discovered. The order a dependency reference is encountered is preserved and the module sorting later performed now honors that ordering.

Benefits of this change:

• Avoids a full parse on every input when CommonJS module processing is enabled.
• ES6 and CommonJS modules no longer generate synthetic goog.require and goog.provide calls.
• ES6 module transpilation no longer depends on Closure-Library primitives.
• CommonJS module rewriting no longer depends on ES6 input

Side effects of this change:

• Multi-threaded parsing will now only apply to externs.

Fixes #2590
Fixes #2535
Fixes #2559
Fixes #2362
Fixes #2247

[thelgevold]

Thanks @ChadKillingsworth
I just pulled down this PR and confirmed that it fixes the original issue described in #2637

[ChadKillingsworth]

@blickly I haven't been able to create a repo case where base.js is ordered incorrectly. I did add a couple of test cases to prove dependency ordering is not dependent on the user provided input order.

The one oddity I found was https://github.com/google/closure-compiler/blob/master/src/com/google/javascript/jscomp/JSModuleGraph.java#L542-L545
While I can't figure out why it would cause a problem, the order of entry points matters. And always adding base.js as an entry point seems "odd".

[ChadKillingsworth]

One other note: historical precedence adds moochers to the front of the entry point list and user defined entry points at the end. If moochers are traditionally tests, should this order be reversed?

[blickly]

I'm seeing cases where this is incorrectly ordering base.js after other files. For example, with the following setup.

// bootstrap.js
goog.require('foo');

// foo.js
goog.provide('foo');
goog.setTestOnly();

When running the compiler with sorting but no pruning with the files passed in the order:
--js=bootstrap.js --js=closure/base.js --js=foo.js
we were previously getting the (correct) ordering of
base, foo, bootstrap,
but after this PR, we're getting an ordering of
foo, bootstrap, base

[blickly]

OK, looks resolved in the current version. Thanks Chad!
Now I'm just seeing issues that need to be fixed on our end...

[alexeagle]

@blickly what's your priority for merging this? Angular can't release our version 5.0 right now because ES6 modules are broken. We are hoping to cut an RC next week and might need to escalate.
/cc @IgorMinar

[MatrixFrog]

We're working on an issue with some internal tests that have done some unusual things.

[alexeagle]

Any estimate of how many tests, and how broken they are? Do you need help to land the CL?

[blickly]

Sure, never hurts to have more help. Thanks!
I've cc-ed you a task on the internal tracker bug.

[Ramblurr]

Could this behavior be cause by this bug?

[ChadKillingsworth]

@Ramblurr Yes definitely

[Ramblurr]

@ChadKillingsworth good to know. I've checked out this PR and while the original error message is gone, I've ran into another one. See the edited portion of the SO question I just added. Could it still be related to this PR?

[lauraharker]

This PR was rolled back in 22240a5 after it broke some internal projects. @MatrixFrog is working on a fix.