Add moduleDetection compiler flag to allow for changing how modules are parsed #47495

weswigham · 2022-01-18T21:26:54Z

The default setting is 'auto', where JSX containing files under react-jsx and react-jsxdev are
always parsed as modules, and esm-format files under module: node12+ are always parsed as modules,
in addition to the 'legacy' detection mode's conditions for other files. (Declaration files are exempt from
these new conditions)

The 'legacy' mode preserves TS's behavior prior to the introduction of this flag - a file is
parsed as a module if it contains an import, export, or import.meta expression.

In addition, there is a 'force' mode that forces all non-declaration files to be parsed as modules.
(Declaration files are still only modules if they contain a top-level import or export.)

This technically breaks the parser API, but it's kinda-sorta backwards compatible so long
as you don't need the functionality associated with more recent compiler flags.

While "auto" is a mostly backwards compatible new default (technically it breaks react-jsx output, but it does so in a case where the output was nonfunctional and fixes it, so it's fine), most people probably just want to use "force" unless they intend to have script files in their compilation.

"moduleDetection": "auto" (the new default):
Fixes #46723
Fixes #40501
Fixes #47237

"moduleDetection": "force":
Fixes #18232
Fixes #14279

…re parsed The default setting is 'auto', where JSX containing files under react-jsx and react-jsxdev are always parsed as modules, and esm-format files under module: node12+ are always parsed as modules, in addition to the 'legacy' detection mode's conditions for other files. (Declaration files are exempt from these new conditions) The 'legacy' mode preserves TS's behavior prior to the introduction of this flag - a file is parsed as a module if it contains an import, export, or import.meta expression. In addition, there is a 'force' mode that forces all non-declaration files to be parsed as modules. (Declaration files are still only modules if they contain a top-level import or export.) This technically breaks the parser API, but it's kinda-sorta backwards compatible so long as you don't need the functionality associated with more recent compiler flags.

sandersn · 2022-01-18T21:51:51Z

Can you add an explanation to the PR description of why 'legacy' is needed -- basically, what you expect to break if everybody were upgraded to 'auto' with no escape hatch.

weswigham · 2022-01-19T01:07:44Z

Hm, I alluded to it, but it would break people with global script files using react-jsx or react-jsxdev with jsx tags in them that somehow were post-processing them to actually work (since they didn't by default, but also weren't an error). Hopefully that's a vanishingly small population.

andrewbranch · 2022-01-24T19:35:13Z

esm-format files under module: node12+

Does this include .ts files when there’s a package.json with "type": "module"?

weswigham · 2022-01-25T07:04:08Z

Yep.

src/compiler/parser.ts

andrewbranch · 2022-01-25T17:42:42Z

src/compiler/types.ts

+         * The callback used to set the external module indicator - this is saved to
+         * be later reused during incremental reparsing, which otherwise lacks the information
+         * to set this field
+         */
+        /* @internal */ setExternalModuleIndicator?: (file: SourceFile) => void;


which otherwise lacks the information to set this field

Why is this the case now but not before this PR?

Because the module indicator now depends on the compiler options the source file was created with, which the source file retains no direct reference to, and small incremental edits (the kind that don't do a full source file rebuild) don't pass in full options, either, they just reuse the old source file's data - so we have to keep the callback around to reuse in that incremental scenario.

Hm, I forgot that parsing in general, much less incremental parsing, doesn’t have access to CompilerOptions; only the (previously very small) subset of them that it needs. I’m a little worried that this will become a footgun whereby we or other API consumers will accidentally close over and retain something they shouldn’t in this function (it you haven’t seen the implementation, you don’t know a reference to it is stored on the SourceFile—the property is even erased from the public API!). What if CreateSourceFileOptions took the moduleDetection, module, and target from CompilerOptions and stored those, such that the function currently returned by getSetExternalModuleIndicator would have everything it needs to run on the SourceFile after creation? It would be more properties to store, but more straightforward. It took me a minute to understand the purpose of this indirection.

Hm, while that would be enough for recalculating the correct closure, it doesn't do much to help the impliedNodeFormat field calculation, which needs extensive host availability anyway. I'm disinclined to make these look like CompilerOptions because of that - if you just passed in compiler options, you'd be broken in any nodenext type scenarios.

impliedNodeFormat now gets passed into createSourceFile (and I wasn’t proposing removing that, for the reason you mentioned), and I was assuming it can’t change during updateSourceFile (incremental parsing)—is that right? If so, then incremental parsing doesn’t need to worry about impliedNodeFormat.

tests/baselines/reference/api/typescript.d.ts

src/services/documentRegistry.ts

src/compiler/program.ts

fatcerberus · 2022-01-29T01:20:48Z

esm-format files under module: node12+ are always parsed as modules

I’m unclear on what constitutes an “esm-format file” if it’s not based on the presence of import/export, seeing as ESM mode is controlled by the runtime, not the file contents. Does this mean node.js semantics, i.e. .mts is always a module, .ts is a module only if "type": "module", etc.?

If the above is the case it sounds like I probably want to use force when targeting ESM in the browser.

weswigham · 2022-01-29T08:45:46Z

Yes and yes. Esm mode modules are the only ones which transpile to esm output, to whit (cjs mode files transpile to cjs). Honestly, if you're targeting browser modules you probably want moduleDetection: force and module: esnext, rather than nodenext, since you don't want cjs in your compilation at all.

sandersn

I think this is a good idea. I made some low-level comments, but I don't understand the higher-level structure enough to make useful comments there.

I'll look at the tests next.

src/compiler/diagnosticMessages.json

src/compiler/parser.ts

sandersn · 2022-03-02T21:47:58Z

src/compiler/utilities.ts

+    /**
+     * This is a somewhat unavoidable full tree walk to locate a JSX tag - `import.meta` requires the same,
+     * but we avoid that walk (or parts of it) if at all possible using the `PossiblyContainsImportMeta` node flag.
+     * Unfortunately, there's no `NodeFlag` space to do the same for JSX.


should there be? (there might not be any NodeFlags left in any case)

This bails on the first JSX tag you find in a JSX tag, which is usually almost instantly, so.... probably not? Still, I think node.transformFlags & TransformFlags.ContainsJsx probably already tracks the same thing and post-node-factory change should always be set, so... I'll try that out for narrowing the search.

src/compiler/utilities.ts

src/services/transpile.ts

src/services/types.ts

tests/baselines/reference/api/tsserverlibrary.d.ts

sandersn

Done looking at the tests, just waiting on answers to earlier questions.

src/compiler/diagnosticMessages.json

nayeemrmn · 2022-03-12T10:37:42Z

In addition, there is a 'force' mode that forces all non-declaration files to be parsed as modules.
(Declaration files are still only modules if they contain a top-level import or export.)

@weswigham Does having a .cts extension override "moduleDetection": "force"?

weswigham · 2022-03-12T11:08:37Z

A .cts module is still a module, just a cjs format one instead of an esm one. The flag on force disables script mode files, which are like, the old global browser global-type scripts.

nayeemrmn · 2022-03-12T11:15:05Z

Ah okay, the script mode stuff doesn't refer to cjs but old-fashioned browser scripts. Thanks.

typescript-bot assigned weswigham Jan 18, 2022

typescript-bot added Author: Team For Milestone Bug PRs that fix a bug with a specific milestone labels Jan 18, 2022

weswigham requested review from andrewbranch, amcasey and sandersn and removed request for andrewbranch January 18, 2022 21:27

weswigham added 2 commits January 18, 2022 13:33

Merge branch 'main' into parser-new-signature

53a409b

Fix post-merge lint

60f709b

andrewbranch reviewed Jan 25, 2022

View reviewed changes

src/compiler/program.ts Show resolved Hide resolved

sandersn added this to Not started in PR Backlog via automation Feb 1, 2022

sandersn moved this from Not started to Waiting on reviewers in PR Backlog Feb 1, 2022

RyanCavanaugh mentioned this pull request Feb 8, 2022

Remove warnings on empty file when enabled IsolatedModules #47800

Closed

5 tasks

weswigham mentioned this pull request Feb 10, 2022

BUG: (edge case) Typescript 4.5+ moduleResolution GlobalScope Introduce var global inside ESM Context edge case #47237

Closed

weswigham added 2 commits February 11, 2022 15:14

Merge branch 'main' into parser-new-signature

9ad5610

Rename function

6215541

sandersn reviewed Mar 3, 2022

View reviewed changes

sandersn requested changes Mar 9, 2022

View reviewed changes

PR Backlog automation moved this from Waiting on reviewers to Waiting on author Mar 9, 2022

weswigham added 2 commits March 9, 2022 10:43

Merge branch 'main' into parser-new-signature

9bd8c47

Update default value documentation

ccb851a

PR feedback

f9eb659

weswigham requested a review from sandersn March 9, 2022 19:18

jakebailey reviewed Mar 9, 2022

View reviewed changes

src/compiler/diagnosticMessages.json Outdated Show resolved Hide resolved

weswigham added 2 commits March 9, 2022 12:42

Merge branch 'main' into parser-new-signature

dcc86cc

Fix lint and typo

e5c3dc0

sandersn approved these changes Mar 10, 2022

View reviewed changes

PR Backlog automation moved this from Waiting on author to Needs merge Mar 10, 2022

weswigham merged commit d1fa945 into microsoft:main Mar 11, 2022

PR Backlog automation moved this from Needs merge to Done Mar 11, 2022

phudson mentioned this pull request Mar 11, 2022

Configuration For Treating JS/TS Files as Independent Units (modules) #14279

Closed

kitsonk mentioned this pull request Mar 12, 2022

Determine correct moduleDetection compiler option (TypeScript 4.7) denoland/deno#13924

Closed

DanielRosenwasser added the Breaking Change Would introduce errors in existing code label Apr 7, 2022

azu mentioned this pull request Apr 13, 2022

2022-04-13のJS: TypeScript 4.7 Beta、RedwoodJS 1.0、Sanitizer API jser/jser.github.io#975

Merged

danvk mentioned this pull request Sep 7, 2022

Update getLanguageServiceHost() to support TS 4.8 JoshuaKGoldberg/eslint-plugin-expect-type#61

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add moduleDetection compiler flag to allow for changing how modules are parsed #47495

Add moduleDetection compiler flag to allow for changing how modules are parsed #47495

weswigham commented Jan 18, 2022 •

edited

Loading

sandersn commented Jan 18, 2022

weswigham commented Jan 19, 2022 •

edited

Loading

andrewbranch commented Jan 24, 2022

weswigham commented Jan 25, 2022

andrewbranch Jan 25, 2022

weswigham Jan 25, 2022

andrewbranch Jan 25, 2022

weswigham Jan 25, 2022

andrewbranch Jan 25, 2022

fatcerberus commented Jan 29, 2022

weswigham commented Jan 29, 2022

sandersn left a comment

sandersn Mar 2, 2022

weswigham Mar 9, 2022

sandersn left a comment

nayeemrmn commented Mar 12, 2022

weswigham commented Mar 12, 2022

nayeemrmn commented Mar 12, 2022

Add moduleDetection compiler flag to allow for changing how modules are parsed #47495

Add moduleDetection compiler flag to allow for changing how modules are parsed #47495

Conversation

weswigham commented Jan 18, 2022 • edited Loading

sandersn commented Jan 18, 2022

weswigham commented Jan 19, 2022 • edited Loading

andrewbranch commented Jan 24, 2022

weswigham commented Jan 25, 2022

andrewbranch Jan 25, 2022

Choose a reason for hiding this comment

weswigham Jan 25, 2022

Choose a reason for hiding this comment

andrewbranch Jan 25, 2022

Choose a reason for hiding this comment

weswigham Jan 25, 2022

Choose a reason for hiding this comment

andrewbranch Jan 25, 2022

Choose a reason for hiding this comment

fatcerberus commented Jan 29, 2022

weswigham commented Jan 29, 2022

sandersn left a comment

Choose a reason for hiding this comment

sandersn Mar 2, 2022

Choose a reason for hiding this comment

weswigham Mar 9, 2022

Choose a reason for hiding this comment

sandersn left a comment

Choose a reason for hiding this comment

nayeemrmn commented Mar 12, 2022

weswigham commented Mar 12, 2022

nayeemrmn commented Mar 12, 2022

weswigham commented Jan 18, 2022 •

edited

Loading

weswigham commented Jan 19, 2022 •

edited

Loading