[RFC] Tooling: scaffolding, bundling, cli helper, support of Aurelia convention.

[3cp]

💬 RFC

Tooling around Aurelia 2: scaffolding, bundling, cli helper commands, and support of Aurelia convention.

🔦 Context

At Aurelia v1's early time, complexity around scaffolding and bundling hurt our early adoption.

Our target is to make sure Aurelia 2 provides smoothest possible tooling for our users.

Current aurelia-cli is mainly designed for Aurelia v1, especially the built-in bundler. While aurelia-cli provides a solid user experience, the all-in-one approach made it hard to add feature, test, release, and slow to install. aurelia-cli is a huge three-in-one: scaffolding, a bundler, and convenient helper commands.

With Aurelia 2, we are going to modularize tooling parts, separate scaffolding, bundling, convenient helper commands. But this is just internal design to us, our end users would not and should not see any difference. The separation of concerns would allow us to improve and iterate each part faster, plus easier testing.

1. Scaffolding

Zero-install scaffolding. This will be enabled by a new GitHub repo aurelia/new, a skeleton-only repo for "makes".

npx makes aurelia

"makes" is a scaffolding tool I designed based on my experience on aurelia-cli. It enables zero-install scaffolding, speed, great flexibility and simplicity, plus easy try-out on feature branch. It's fully documented, read more if you are interested.

2. Bundling

Different from Aurelia v1, Aurelia 2 has no loader abstraction layer. This simplification has made Aurelia 2 naturally working with any bundler without any effort. We got few examples in Aurelia 2 repo.

The simplification comes with a price. With Aurelia 2, users would need to use explicit decorator to mark custom elements, custom attributes, value converters, and binding behaviours. This almost made Aurelia 2 code as verbose as Angular. But we have a plan (explained in 4. Aurelia convention) to bring Aurelia convention back through our tooling. In the end, end users will see Aurelia 2 uses convention to cut out boilerplate code just like Aurelia v1 did.

Let's keep Aurelia 2 setup clean. Let webpack be webpack, don't use gulp + webpack. Let browserify be browserify, keep it as simple one line command in npm scripts if possible.

The scaffolding and bundling will provide standard npm scripts.

• npm start to start dev server.
• npm test to run unit tests.
• npm run build to build production files.

What about aurelia-cli built-in bundler

For users of aurelia-cli built-in bundler, they probably want to keep using an AMD module bundler. We got a better implementation.

dumber bundler is the successor of cli-bundler. The redesign solved many architecture problems of existing cli-bundler. For example, dumber fully supports CommonJS circular dependency which finally brings npm package compatibility to the level of browserify/webpack. Technically, dumber is better than cli-bundler, simpler and more flexible, plus great test coverage which is a weak area of aurelia-cli code base.

I still need time to finish the dumber documentation, but I am running all my production apps with dumber. We will provide dumber bundler as one of the options (others are webpack/parcel/browserify/fuse-box).

Although current cli-bundler can boot up Aurelia 2 app, we are unlikely to ship an official skeleton with it. Because 1) majority of aurelia-cli code are not needed by Aurelia 2; 2) au generate needs a new implementation for Aurelia 2 anyway, we don't want to push this new feature into existing huge aurelia-cli code base.

3. CLI (To be discussed in future, whether we need it, or its scope)

CLI provides helper commands. It's totally optional for Aurelia 2 app. Experienced users can opt-out CLI for lean setup.

We will keep Aurelia 2 CLI small and flexible. To avoid conflict and confusion with existing au command for Aurelia v1, we will name the new CLI au2.

• In the core, au2 only implements various au2 generate helper commands.
• au2 new simply calls npx makes aurelia.
• au2 test and other commands simply call standard npm scripts like npm test.
• (To be decided) Since users need npm i -g au2 to get au2 command, there is no need to install au2 package in app itself. This will help to keep app lean.

4. Aurelia convention

Aurelia v1 has strong beloved convention, which is used by aurelia-loader at runtime to load various resources (html template, custom element, value-converter, binding-behaviour). Since Aurelia 2 has no loader abstraction, there is no such convention baked in. Aurelia 2 needs explicit decorator/code to hook up resources.

Out of scope of this RFC, Aurelia 2 AOT (Ahead of Time compilation) is yet to be implemented by @fkleuver. That will take quite some efforts. In the end, AOT will implement all Aurelia convention, plus great deal of tunable optimizations. But we are not going to wait for AOT in order to ship Aurelia convention in Aureila 2.

There is a cheap path to implement convention through tooling. We can mutate source code (inserting all necessary decorators) based on convention, before bundler ever packs the code.

Here is a working POC for only implementing @customElement convention through gulp for dumber bundler. It allows user to write js/html pairs just like Aurelia v1 without using explicit @customElement decorator (well, the gulp task write the decorator for you).

We can take this idea to implement Aurelia convention for all scenarios of custom elements, custom attributes, value converters, and binding behaviours. This will enable users to write Aurelia 2 without boilerplate just like Aurelia v1, even though Aurelia 2 doesn't have convention implementation baked in runtime.

Plan

• Create aurelia/new repo to provide basic Aurelia 2 setup.
  • Support webpack, parcel, browserify, fuse-box, dumber.
  • Support jest, jasmine, mocha, tape.
  • Might skip karma, and go with browser-do for simpler browser testing.
  • At this stage, we will leave e2e setup to future offering.
• Base Aurelia convention implementation @aurelia/plugin-conventions.
  • Wrap around @aurelia/plugin-conventions to create plugins for webpack, gulp (dumber uses), and parcel, browserify, fuse-box (these three were put on-hold).

The above wouldn't take too long, I guess less than 3 months of my spare time. They can provide early adopters a smooth start. Unit tests might depends on maturity of @aurelia/testing.

The coming aurelia/new skeleton repo will use plain CommonJS code, because that's the design decision of "makes" which tries make skeleton simple without any runtime dependencies.

The future will be AOT but it is out of scope of this RFC.

Let's give early adopters a great landing experience.

[HamedFathi]

@3cp
Thanks,

May I ask you to make new CLI extendable?
Sometimes users/developers need to make a plugin for CLI.

[3cp]

What kind of plugins you are thinking?

Some plugins can be bundler plugins.

[3cp]

The au2 CLI is far on the road. We are not designing it at this stage, it is just a vague idea around it.

At start, we will deliver a simple scaffolding with various bundler setup for Aurelia 2 without CLI.

[HamedFathi]

What kind of plugins you are thinking?

I am looking for something like Vue CLI Plugins and Presets

How to build your own vue-cli 3 plugin

For example: Making Tailwind CSS as a CLI plugin instead of making PR on CLI itself.
I believe that with such a feature, CLI can be extended by Aurelia`s community.

Sometimes the requirements are more limited than adding to the original project as a feature (Aurelia CLI), but it can be useful as a person or team capability. (as a CLI plugin)

The au2 CLI is far on the road. We are not designing it at this stage, it is just a vague idea around it.

Yes, I know. This is just RFC. No problem.

[3cp]

Oh, scaffolding is out of au2 CLI. Scaffolding will be purely handled by repo aurelia/new (to be done).

I'll have a look at vue cli plugins, to see if there is something we can do to enhance the extensibility.

[3cp]

Vue CLI has less trouble to extend, because they only ship webpack setup (correct me if I am wrong).

But we want to support quite a few bundlers, everyone has different setup, this will add complexity when let's say to add tailwind css to the scaffolding.

[3cp]

The vue cli presets are handled by "makes" silent mode.

I sensed some trouble on vue cli plugins. The plugin tries to mutate the base skeleton (vue cli default offering), but that means the vue cli default skeleton needs to be very stable over time to not break 3rd-party vue cli plugins.

[HamedFathi]

Vue CLI has less trouble to extend, because they only ship webpack setup (correct me if I am wrong).

Yes

But we want to support quite a few bundlers

I believe that a good choice with full development possibilities is far better than several choices with weaker ones.

Many users of Aurelia do not comment on the way CLI works and what are bundlers. They just want to develop their project in the most convenient way.
Different bundles are important for more professional programmers and I think eventually they can configure their own projects.
If we support one approach well, we will achieve our goals. The multiplicity of choices reduces the conditions for developing and meeting needs in the future and makes Aurelia CLI's development harder.
By choosing a suitable method, we can extend the development of the CLI to the community. Certainly we can not be ready for all needs.

Of course, this is just my opinion.

[3cp]

I understand. Because of the market share, webpack will be our default bundler if I am not wrong. We can ship different feature set for different bundler. We can ship rich feature set for one or two bundlers, while keep other bundlers setup lean and simple.

Like current aurelia-cli, if users choose one of "default", not "custom", they will get a default webpack app. The question of bundlers will not be asked.

"makes" is flexible enough for conditional features.

If we want to keep default skeleton repo as clean as possible, we can also split skeleton into two repos aurelia/new and aurelia/custom.

npx makes aureila # use aureila/new, it's webpack only
npx makes aureila/custom # provide choices of other bundlers

[HamedFathi]

Because of the market share

Exactly

webpack will be our default bundler if I am not wrong

I am not a fan of Webpack but you are correct. It seems we have to!
I prefer Fusebox but unfortunately, they don't support Bundle analyzer yet 😑😔

We can ship rich feature set for one or two bundlers, while keep other bundlers setup lean and simple.

Yes, Good idea. Even we can support one approach and release some articles about how to support others.

Thank you for hearing me. 😊

[3cp]

Great, let's keep skeleton to single repo for now.
Two or more skeleton repos will impose some duplicated files to be synced.

[EisenbergEffect]

Thanks for putting this together @3cp ! I’m wondering, is there any way we can keep our skeletons inside our main monorepo? Would makes be enabled to handle that or do we need a separate repo? It’s ok if we do, I just wanted to check.

[3cp]

It's possible, because I can change/enhance "makes".

But I would suggest to keep scaffolding skeleton out of lerna because it's not a npm package to be published to npm, it's just a group of files. "makes" supports simple skeleton from git repo (even private) or local folder. Nothing to be published, hence always "latest" release. To test an unreleased feature branch, just do npx makes aurelia/new#feature. We only need to protect master branch from untested features, to take advantages of the simplicity.

I think merging them will make things complicated, both the lerna side and "makes" side. It will certainly complicate "makes" little bit to look into a deep folder. But I am open to ideas.

[3cp]

Ok, I got a good reason to reject the idea.

At runtime, because of zero-installation, "makes" needs to download a tar.gz ball or clone the repo to get all the files.

I think it's not practical to download such a big repo, then only use a small part of it :-)

[EisenbergEffect]

That makes sense, and I’m quite fine with having a dedicated repo for this purpose.
I like this general plan. @3cp What are our next steps?

[EisenbergEffect]

Oops. I Should look back at the issue :) @3cp I guess the next step is for me to create the repo. Does it just need to be named new?

[3cp]

Yes. I can get first skeleton up in few days, without convention, without unit test.

[EisenbergEffect]

Alrighty, @3cp you are all set up here: https://github.com/aurelia/new

[michaelw85]

This is very interesting and I would like to help if possible.

I would like to suggest to add the ability for the skeleton to either pin dependencies or ship with a package lock. I've seen others complain about a broken setup and sad to say this was also my first experience with Aurelia due to bugs in deps.

[michaelw85]

A nice to have might be the ability to enable features after you have setup a project. In case someone start very lean but requires more after all or you want to switch features, like unit test framework.

[3cp]

Thx! At this dev stage, we don't want a lock file, because we want to test all Aurelia 2 daily dev version release. Things will change when we started to ship Aurelia 2 production build.

But you can start to experiment lock file shipment now.
The skeleton uses "after" task to install deps, you can enhance the task to ship a lock file based on user selected features, then maybe use run('cp', [absolute_lock_file_path, '.']) (and something equivalent on win32) to get the file into the final app.

With the lock file, I think there are few responsibilities you need to think of:

1. Build the lock file for every possible combinations of select features, based on e2e test result. Ideally, the same lock file has to pass e2e test on all win32/linux/macos.
2. Some automation to build all the lock files, check them in.
3. Everytime Aurelia 2 release a patch or minor version, somehow an automatic bot will test the new version and generate new lock files.
4. I guess it will be a manual update when a 3rd-party dep released a patch or minor version.

With or without lock files, we definitely love to have some kind of automatic e2e tests daily to proactively alert us about failed scaffolding.

[3cp]

the ability to enable features after you have setup a project

It's very tricky. You don't know how users changed things around after they created the app. If they didn't touch the standard setup, you might have a chance. But if they didn't touch anything, it's equally easy for them to create a new app, and copy source code over.

[michaelw85]

@3cp Thx for the info, in reply to the lock file. I was thinking along the same lines although the idea of having to build every possible combination + test it kinda scares me in term of processing time/resources/performance. I think the easiest way out is to go with pinned versions at some point.

On the feature part, I agree it could be near to impossible.
I also went the same route as you described, create a new project than copy files for the new feature I wanted. This is doable but far from optimal, it's hard to figure out what is needed if you are unfamiliar with the code or Aurelia. I ended up getting things working but i think it would have been quicker to just implement it myself. I would really like to void this experience for new users.

Here's a brain fart I had this morning:
The current setup described a skeleton that has conditional files (features) that will be combined into a finished project to use. This provides a lean setup with just the files your need but complicates the skeleton/scaffolding (although you made it as simple as possible with "make").

What if there is just a single skeleton that includes all features, no conditional files. This would make the skeleton very simple to maintain and remove the need for any additional tooling apart from copy files from a -> b.
We could use tooling like nps as a proxy layer to determine what commands to run based on feature toggles or make the cli responsible.
Switching in terms of what feature you use would just be a configuration toggle + installing package deps.

[3cp]

But that is not only complex to build, it also give user an all-in-one solutions for everything which is what we desperately try to avoid for aurelia 2.

We think users appreciate more on lean app.

But makes give us the opportunity to ship different skeleton for maybe beginners. Just create another repo aurelia/easy.

npx makes aurelia/easy

[michaelw85]

On the skeleton part did anyone look into existing tooling like Yeoman for example instead of building custom tooling? Any reasoning behind going custom?

[3cp]

Try to create a skeleton following yeoman doc.
Do the same thing follows makes doc.
Then you understand why I created makes.

[3cp]

@EisenbergEffect question on existing plugin-requirejs implementation.

First, a small problem. It supports import "some.html"; inside html template which is outdated syntax, replaced by <import from="some.html"> which is implemented in plugin-conventions.

Do we still need this runtime requirejs plugin? I can build a thin plugin-gulp to preprocess html template to code at bundling time, that will work for both requirejs and dumber.

In long term, for runtime dynamic loading, I think we should normalize that on top of native import() API (or maybe fetch() for html file, anyway both import() and fetch() are promise), so same app code can work in any bundler. The plugin-requirejs only solves dynamic loading in requirejs which is too narrow and requires user to write import template from 'view!./foo.html'; which is not portable.

[EisenbergEffect]

@3cp I'm in agreement with you. I think we can delete that old require.js plugin. It doesn't actually work anymore and it was a very early experiment anyway.

[3cp]

Ok, I will remove it when I submit plugin-gulp PR.

[adriatic]

This RFC looks great - where I have no idea how close is the initial implementation of Aurelia 2 tooling. Being interested in getting a very preliminary preview (I want to try building WordPress with Aurelia front end), I glanced over the docs and using the latest update of the user-docs.

I think that the document Building and testing Aurelia is intended for curious people like me. I also read most of the Engineering Notes and was able to build Aurela 2, without a hitch.

Lastly, I am aware of the few existing examples - and will appreciate some minimum guidance on (not yet existing) section on "local deployment" of Aurelia 2, so that these examples would be "meaningful".

Is the Release Procedure the document I could use for my private (local) release?

[EisenbergEffect]

@adriatic You should be able to take any of the examples in the examples folder and copy it somewhere to use as a starter project. They are basically skeletons. After you copy it, do an npm install and then look in the package.json to see what scripts it supports. Most likely you will run npm run watch to run the app in dev mode and npm run build for production build. They each have a readme with instructions that should explain that. We can add a readme to the examples folder itself as well with a general explanation for future explorers. I'd recommend that you use the Webpack setup.

[EisenbergEffect]

@adriatic Could you open a separate issue for this? It's very off topic from the RFC.

[adriatic]

Yes, it is off topic - thanks for addressing this. I created a new issue: Building Aurelia 2 examples collection and will resolve my question by understanding the content of Aurelia 2 Scaffolding skeleton. I remember you mentioned the newly adaption of the makes tool by Chunpeng Huo.

[EisenbergEffect]

I should have chimed in on a particular point earlier: I really don't like au2 as a tool name. I don't want "2" or any version indication to be found in any names themselves if we can help it. It's particularly important in this case since it's only a temporary designation while we go through the transition.

I also think that if we want a CLI, we need a lot more detailed discussion on what that does. It's not completely clear to me why it's needed. It seems to me that we can just stick with npx makes aurelia and then makes can generate package scripts to handle everything else. There's no reason why it couldn't enable npm run generate custom-element for example, leveraging a small generator tool (maybe even sharing code with makes). I don't know that there's justification for a special CLI or separately installed global tools.

[3cp]

Agree, makes can indeed be used for generators too. Also suggest to leave the discussion of generators/helpers to much later time around beta release.

[3cp]

The items in this issue are implemented. Aurelia 2 CLI will likely to take a new issue for its future design.