Plugin Support for Custom Transformers

[MichaReiser]

Since #13764 has landed, it's easy to write custom transformers. However, if I understand the API correctly, it is needed to duplicate the whole tsc command line just to add a single transformer. This leads to incompatibilities since these, on typescript depending, applications do not support the same features as the tsc command line tool.

It, therefore, would be favored to have a plugin system that allows loading Custom Transformers from third party node modules as this is the case for the language service proxies #12231. These transformers then easily integrate into existing build tools / build workflows.

If someone experienced has inputs on how to implement the changes, I'm willing to create a PR as it would simplify my project tremendously.

[mhegazy]

We do not plan on exposing compiler plugin system in the short term. The transforms are exposed as part of the public API as you noted, and we are in the process of writing documentation and samples for these.

[MichaReiser]

I don't want to have an exposed plugin API. Instead, I would prefere way to register my custom transforms by just specifying them in the tsconfig.json instead of having to use the TS-Compiler API. Because using the TS-Compiler API implies that I can no longer use the tsc command line tool which further can imply that it does no longer integrate nicely into existing build tools and workflows.

[WanderWang]

anything update about documentation and samples ? @mhegazy

[longlho]

@MichaReiser you can write a simple compiler based on the API (sample here). We actually use TS very heavily within Yahoo Finance and the transformer public API have been awesome.

Those are some of the transformers we wrote after it became public:
https://github.com/longlho/ts-transform-system-import
https://github.com/longlho/ts-transform-img
https://github.com/longlho/ts-transform-react-intl
https://github.com/longlho/ts-transform-css-modules-transform

@mhegazy lmk if you need help documenting/gathering samples

[MichaReiser]

@longlho
Thanks for your sample.

That is what I'm currently doing. However, it makes it impossible to use other build tools created for typescript, e.g. web pack loaders, jest loaders. Besides, how can I use one of your plugins together with one of mine when each of us uses a different frontend?

Therefore, I do believe that the current approach is a good start but not sufficient for a plugin ecosystem. Because using a plugin is too much effort for the user and requires more than just writing the transformation code for the author.

I believe a plugin system like the one of Babel is essential for typescript if the goal is to encourage the community to create and use custom transformers.

[longlho]

@MichaReiser yup. I'm not saying it's sufficient for the ecosystem, just good enough for the 1st step towards it.

[ivogabe]

I'd like to add support for transforms in gulp-typescript, but I want to prevent that all TypeScript plugins (for gulp, webpack etc) propose a different API. And TypeScript might add even a different way for configurating this later on. So do you currently have plans to add this in the near or far future?

[RubaXa]

Hi all, I try to write a preprocessor, but nothing comes out :[

In fact, I have two questions:

• How to create AST-fragment from string?
• How to add import?

// 1. Input
class Foo {
    templateString = 'some value';
}

// 2. After transformation
import __LIB__ from '@external/lib';

class Foo {
    templateString = (function compiledTemplate(deps) {
        // ...
        return result;
    })({lib: __LIB__});
}

// 3. Expected result
var lib_1 = require("@external/lib");
var Foo = (function () {
    function Foo() {
        this.templateString = (function compiledTemplate(deps) {
            // ...
            return result;
        })({ lib: lib_1 }); 
    }
    return Foo;
}());

A Simplified Example: https://github.com/RubaXa/typescript-api-questions/tree/master/import-add

[RubaXa]

⬆️ ⬆️ ⬆️
@longlho, @mhegazy Can you give any hint?

[rbuckton]

@RubaXa Since you added the import declaration in a transform, it isn't bound or type checked. Since it wasn't bound or type checked, we cannot resolve __LIB__ in your expression to the __LIB__ in the declaration.

One option is to use a namepace import instead of a default import, so that your emit is something like:

import * as __LIB__ from "@external/lib"

As there is no aliasing that can occur.

The other hold onto a generated identifier for the import declaration, as per the attached zip

[rbuckton]

@RubaXa if your goal is to pass in the entire module object, you probably want to use the import * as __LIB__ from "@external/lib" syntax (which does not require aliasing) and not import __LIB__ from "@external/lib" as the latter imports the default export rather than the entire module object.

[longlho]

yeah we're doing import * as foo in our CSS modules transformer as well to prevent aliasing. Although it'd be nice to tap into that to inline certain named exports

[RubaXa]

@rbuckton O, thanks a lot!
Still interested in how to create an arbitrary AST-fragment from string?

function createFragmentFromString(code: string) {
  // ????
}

function visitPropertyDeclaration(node) {
	if (ts.isIdentifier(node.name) && node.name.text === "templateString") {
		// ...
		return ts.updateProperty(
			node,
			ts.visitNodes(node.decorators, visitor),
			ts.visitNodes(node.modifiers, visitor),
			ts.visitNode(node.name, visitor),
			ts.visitNode(node.type, visitor),
			createFragmentFromString('(function compiledTemplate() { /* ... */ })()')
		);
	}
	return node;
}

[rightisleft]

This is what i was hoping the transformer support and workflow would behave like in TypeScript.

https://www.dartlang.org/tools/pub/transformers

[chandu0101]

I am also very much interested in this feature.

as @MichaReiser mentioned
If someone experienced has inputs on how to implement the changes, I'm willing to create a PR as it would simplify my project tremendously.

love to see inputs from experts/typescript compiler DEV .

[MeirionHughes]

Looks like someone has wrapped typescript to expose this functionality. https://github.com/cevek/ttypescript

Also looks like ts-loader has it: https://www.npmjs.com/package/ts-loader#getcustomtransformers-----before-transformerfactory--after-transformerfactory----

[jcimoch]

I think this is something that could land in typescript 2.8 release or at least in roadmap at some point, @ahejlsberg @mhegazy @DanielRosenwasser what do you think? In this way custom transformers might be more popular and therefore more powerfull. Having option to plugin in transfromer from tsconfig.json perspective would simplify life a lot.

[mhegazy]

We have no plans to expose plugins in the short term, as noted earlier.

[raveclassic]

@mhegazy Is it a considered decision or is it out of scope just because of low community interest?

[vsinghipcium]

+1

[adonespitogo]

+1 Badly needed for my intended project.

[truemoroz]

We need this

[mindplay-dk]

Given the huge list of projects already using this, you would think it's time to put it on the roadmap. 🤔

And this list is just projects using it specifically for reflection - another neglected feature many people miss bad enough that they'll go and build their own solutions; even with no official support for building such a feature, meaning every single one of these projects "hack" this feature in somehow.

I understand the list of requested features remains huge, but these are clearly two of the most sought-after features. You would think someone on the team would take note.

If they were to officially open up the language to third-party extensions, I'm sure a lot of these features would get built by the rather large community already trying to "hack" them in without a sanctioned, stable API.

[lorenzodallavecchia]

No one has mentioned yet that Angular is forking the entire typescript compiler to add AOT compilation of templates.
Angular users are, therefore, locked-in to a specific TypeScript version.

Curiously, Angular folks have put the rewrite of the Angular compiler as a TS plugin on their roadmap.
https://angular.io/guide/roadmap#improved-build-performance-with-ngc-as-a-tsc-plugin-distribution

[maheshbansod]

Would be awesome if we had this in our project.
How can I check the progress on this?
would love to help out if I can.

[thislooksfun]

I am looking to do some comple-time massaging of the exported types from my library so I can massively simplify its development. When investigating options I found that the tsconfig has a plugins option and thought "oh good, it's gonna be easy, I just need to write a plugin", but when I looked further for how to actually make said plugin, I found this issue.

You already have support for it in the compiler API, you already have the plugins option in the tsconfig, and people have already done the work of joining the two together (just see ttypescript and ts-patch).

Would the TS team be open to PRs for this? There is enough support for this feature that we can definitely make the actual implementation happen as long as the TS team will receive it.

[DrewRidley]

Please seriously consider this! There really is no harm in supporting this as far as I can see, and implementation would be very simple. All it really requires is calling each plugin during transformation!

[samchon]

As a developer of typia and nestia utilizing transformation, whenever there is a critical patch for the TS compiler API, the libraries break and stress me out.

Quietly vote yes and go.

[gerardmarquinarubio]

When this issue was opened I was 16 years old and just started using TypeScript, this problem went way over my head and didn't understand what it was about.

I'm now 23 and been following this thread for a little bit more than 6 years. The question is: Will my kids be able to write transformers in an easy manner without depending on external wrappers like ttypescript?

[codeninja-ru]

bump

We need this feature

[muhamedkarajic]

We need this feature :)

[t2wei]

we need this feature, thanks!

[sunrabbit123]

The lack of an official transformer means that users will have to create and use their own.

But every time a version of Typescript goes up, the libraries suffer and need to be updated.

Among them is ttypescript, which is widely used, but whose creators don't care.
For now, we support TS@5 with the same usability via yarn add ttsc. this package

The point is that the lack of an official transformer is an inconvenience for many users and slows down the evolution of the language's ecosystem.

[sosoba]

Hello @sunrabbit123

Your documentation shows an incomplete type declaration for the plugins:

(program: ts.Program, config?: PluginConfig) => ts.TransformerFactory;

In fact there is a third argument which is important.

(program: ts.Program, config: PluginConfig, helpers: { ts: typeof ts; addDiagnostic: (diag: ts.Diagnostic) => void }) => ts.TransformerFactory;

Passing references to ts makes the plugin completely dependency-free.
Passing callback addDiagnostic allows it to interact with the user in accordance with the compiler standard (instead of console.log).

I think it's important that authors of their own compilers and plugins maintain this standard.

[sunrabbit123]

@sosoba
Thank you for reporting the issue.

We'll move it to an issue for decontextualization.
Thank you.

solar-religion/ttsc#4

[samchon]

@jakebailey

TS 5 update is shutting down all transformers based libraries for one month. It's a recurring problem whenever TS version be updated. Currently, @nonara (author of ts-patch) is working hard for supporting transformer libraries, but such problem may repeated in someday.

It's been a while since this issue was opened, and we know that these requests are repeated all the time. What are the barriers to supporting official plugins option in TS? If it is because of the lack of documentation or guides for the TS Compiler API, would you be willing to settle it as an official specification if I help you document this part?

p.s) Currently, the author of STC has promised to provide plugin API. It's just a pity that the official TypeScript compiler doesn't have the plug-in capabilities that the unofficial TypeScript compiler provides.

[jakebailey]

Assume the answer were "yes", let's add custom transformers, ignoring all previously-presented reasons not to do it (this is a long thread I have not read in totality). Note that this issue is not "compiler plugins" (#16607) which is a massively less approachable idea compared to the very well-defined idea of "just allow extra transformers".

These are the challenges I see so far:

• Adding plugin support similar to the language service plugins will prevent tree shaking in tsc.js. This will grow the file by 200KB and increase tsc's startup time by 10%. (200KB feels low; per 25a85d1 this should be a lot more, and I'm not sure why it isn't as bad now as it was then. EDIT: doh, typingsInstaller is the problem. That's not a big deal to work around.)
• The "services" project is not included in tsc.js, so the API familiar to the users of typescript.js would not be available. This means that:
  • We could include the services project in tsc.js, increasing its size by another 2MB or so. This will slow things down even more, both by the size increase, but also a shift in the "object allocator" that creates Nodes/Types/Symbol/etc. IIRC, the allocators are poised to go away in the future anyway.
  • We could come up with a new d.ts file for tsc plugins which describes the limited API available in the core compiler. This seems more reasonable (a la tsserverlibrary), but is more complexity.
  • We'd need to decide if the the "deprecatedCompat" code would be in tsc.js. If we don't do this, we can't deprecate anything safely, as a plugin may depend on something we want to remove. If we do do this, then the compiler may be slowed down by the hotpatching the deprecation system does. The latter is certainly not a good idea without ensuring performance is maintained.
• Like LS plugins, these plugins could only be CJS thanks to the synchronous nature of the codebase, unless we majorly restructure. "Async in TS" is a totally different can 50 gallon barrel of worms.

If it is because of the lack of documentation or guides for the TS Compiler API, would you be willing to settle it as an official specification if I help you document this part?

I don't think this is in any way blocked by documentation. If you believe there are holes in our current documentation, those would totally benefit from a dedicated issue. See also #53239.

[nonara]

@samchon There are valid reasons for why the team chose not to do this, many of which Jake detailed above.

Regarding keeping up with TS changes, part of the reason it took longer for me to do this time is because it was time to fundamentally change the underlying structure to make what we're doing easier. It was long overdue.

So, not only will keeping up be easier in the future, we will ultimately be adding CI to monitor dev builds. This will help us add support for new versions before they're released.

As a side note, I would like to kindly ask that people not post issues on the TS repo that pertain to ts-patch. I recognize that your comment here is more broad and applies to this discussion, so I'm more referring to some of the other posts that have been made recently.

Namely, I've been tagged a few times in complaints directed toward the TS team about changes that broke tsp. I'd like to remind everyone that the onus is on us to keep up, and Jake has been kind enough to even let us know ahead of time when big changes happen.

TLDR is - Right now, you can use beta3, which is stable. In the future, we are working to better avoid gaps between supporting new versions.

[jakebailey]

I have just filed #54276, which is a concrete / minimal proposal for closing this issue. In short, the proposal currently being evaluated simply adds the ability to provide CustomTransformers (with access to Program), using configuration / an API similar to ttypescript or ts-patch.

I expect that we'll probably close this long thread at some point; it's gotten so unwieldy and large that I'm not sure that it's very useful anymore. If we do #54276, it will not preclude any new features or additional extensibility of tsc, and I think that addons can be discussed in new issues (or, if in scope for #54276, on that thread).