[Composability] Generators composability

[SBoudrias]

Hi there,

I would've post on Yo.next repo, but issues are closed over there, so...

Anyway, we spoke about how 2.0 should allow better composability between different generators. There wasn't any clear way of possibility, so we should brainstorm over how generators could work togheter.

---

I was thinking about the issue, and I felt like an intent API (like web intent or the android core intent system) could make sense in this case.

Let's say Angular generator generates a controller and would like to scaffhold a test. It could trigger an intent like .intent('test', { filePath: 'controllers/foo.js' }) and then other generators who're listening to the test event can respond accordingly. Of course test is an easy one, I can see how some types of intent could be harder to define... But it is a way of specifying standard communication protocol between multiple generators.

[wesleytodd]

+1 for this. I messed around with composing generators a few weeks ago and it was painful with the current api offerings. The idea of intents seems perfect. I think it would be good to have a menu display if multiple generators are registered for the same intent, something like:

Which test generator would you like to use?
1) Jasmine
2) Mocha
3) Karma

Input number: 

[btilford]

+1
I like that idea.

On Sun, Sep 15, 2013 at 3:00 PM, Wes Todd [email protected] wrote:

+1 for this. I messed around with composing generators a few weeks ago and
it was painful with the current api offerings. The idea of intents seems
perfect. I think it would be good to have a menu display if multiple
generators are registered for the same intent, something like:

Which test generator would you like to use?

1. Jasmine
2. Mocha
3. Karma

Input number:

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/345#issuecomment-24480074
.

[addyosmani]

To follow up, we're going to put together a design doc for composability very soon and establish a timeline for getting this in place. It's definitely the next big step for the project and is important to us.

[SBoudrias]

Still vague ideas, but I'll throw them here

Queueing

Follow up after a concerns in #162 brought up an idea about managing queues (like Backburner.js).

Composable generators could simply be called once, and they'd register all of their actions in a queue.

For example, a Generator-Angular could specify it is compatible with a test generator:

this.allowComposability('test', {
  default: 'generator-karma'
});

Once this method is triggered, the generator looks for a matching generator (maybe ask which available test generator to use on the prompt), then runs it. When running, Generator-karma simply add his required methods to relevant queues. This way they are both run in symbiose, not one after the other.

[addyosmani]

@SBoudrias I like the idea. Could you flesh out further ones for composability support in the Google doc we have going? Should have a link to it from todays email :)

[SBoudrias]

Yeah, I wasn't sure to write directly in the Google docs as it is a more formatted doc, and I just want to throw idea around for now. I'll try to put my mind together and make something more consistent.

Command

Also, an issue I see with composability is the case where two generator edit the same file (Gruntfile) and create a conflicts. This issue can't be resolved with actual "template" solution to fill file content.

A potential solution could be to extract away the common file generation to the generator system.

this.gruntfile.addTask("grunt-contrib-compass", {
    compass: {
        options: {
            sassDir: "assets/sass"
        }
    }
});

In a similar fashion, common configs could be held by the environment and shared between generators. For example, paths as to where the test folder is, etc.

[simonfan]

Guys, I've been fiddling with the idea of generator inheritance and extendability and have just put up some stuff, but haven't actually tested the idea.
The idea is to allow generators to inherit other generators' subgenerators through invocations.

As generators' subgenerators are inferred through directory structure

generator-<%= generatorName %>/     // root
    app/                                // app-subgenerator (main) (generator:app | generator)
        index.js                        // runs app-generation logic
        ...
    <%= subgeneratorName %>/            // subgenerator-subgenerator (generator:subgenerator)
        index.js                        // runs subgenerator-generation logic
        ...

We could emulate that directory structure and create "proxy-generators" that invoke other generators and do stuff after the invocation is done..

generator-express/
    app/
        index.js            // scaffolds what express(1).
        templates/

inherit generator-express:

generator-express-base/
    app/
        index.js            // invokes 'express:app'
    route/              // creates a route
        index.js
        templates/

inherit generator-express-base:

generator-express-rest/
    app/
        index.js            // invokes 'express:app'
    endpoint/
        index.js            // invokes 'express-base:route' + endpoint specific tasks
        templates/
            _endpoint.js

inherit generator-express-base:

generator-express-markdown/
    app/
        index.js            // invokes 'express:app'
    markdown-server/
        index.js            // invokes express-base:route + markdown-server specific tasks

inherit generator-express-markdown AND generator-express-rest:

generator-express-app-server/
    app/
        index.js            // invokes 'express:app' + app specific tasks
    markdown-server/
        index.js            // invokes express-markdown:markdown-server
    endpoint/
        index.js            // invokes express-rest:endpoint

That way, "module generators" could be very specific on what they do, allowing for "app generators" to "inherit" multiple "module generators" functionalities..

Again I have not thoroughly tested the idea, but so far it seems to work... what are your ideas?

[SBoudrias]

@simonfan This seems good for extending an existing generator, but I think we're trying to find a solution to compose generator who'll work the other side.

I think the problem with the idea you're presenting is that it's easy for an author to customize/mixin an existing generator into its solution, but it is hard for a end user to combine with unrelated generators (what if they want to use express-markdown + less bootstrap generator?)

Though, that got me thinking looking through filesystem for commands and sub-generator may not be optimal for user who'd want to extend an existing generator. Maybe instead author should manually register commands on the environment.

module.exports = {
  app: require('./generators/app'),
  controller: require('./generators/controller')
}

Then author wishing to extend an existing generator with their new commands or overwriting an existing one could simply extend it:

module.exports = _.extend(
  require('express-generator'),
  {
    app: require('./generator/app')
  }
);

I'm not sure this would really be useful though - and it'd concern me about backwards compatibility.

[addyosmani]

This would break backwards compatibility as far as I can ascertain. That said, I do like the general idea of being able to easily extend generators in this fashion. Would it be possible for us to add more of these suggestions to the design doc for composability we have going at the moment?

[csantanapr]

I created a generator yo dude that its based on other generators

It's mainly to create Mobile Web Apps using Apache Cordova. User first selects one web app generators from a list (dapp, webapp, angular, mobile, polymer, backbone, jquery-mobile) then it runs the generator-cordovacli

In short dude is a generator created of two other generators.

I'm having difficulty on the unit testing side to pass parameters to the other generators from the the main one.

Would the new workflow help make it easier to compose generators of other generators and take testing into account?

Or this is an invalid use case for this workflow?

[SBoudrias]

@csantanapr This is exactly what we strive for.

[rvangundy]

Wanted to toss up a couple thoughts on this.

Thought 1: End user configurability

It would be nice to allow end users to configure of their personal generator environment without getting in to the existing generator class and inheritance pattern. To be strictly analogous with Grunt (this is only for high-level conceptual purposes):

yeoman.initConfig({
  angular : { // some config options },

  deployPack : {  // This is a reference to a hypothetical generator add-in I'm considering...
    heroku : true,
    ftp : true
  },

  coffee : {
    usePrompt : false  // Ignore prompting and force coffeescript conversion
  }
});

yeoman.registerGenerator('myGenerator', [
  'angular',
  'coffee',  // A hypothetical generator add-on that converts everything to coffee script
  'deployPack'
]);

yeoman.registerGenerator('myGenerator:model', [
  'angular:model',
  'coffee'
]);

Then the user can call their own generator with yo myGenerator and yo myGenerator:model.

Thought 2: Functional composition vs. classical composition

Rather than consider all the ways you can mix in Generator classes, maybe simplify the interface a bit. For a given generator script, make it possible to export a simple function rather than the generator constructor. Make the only parameter to that function a root generator, like so:

module.exports = function(generator) {

  // Push new prompts on to the generator
  generator.prompt(// some new prompts);

  // Add some new algorithm for working with the current context
  generator.use(function(ctx) {
      var prompts = ctx.prompts;
      var fs = ctx.fileSystem;
      var pkg = ctx.pkg;

      // Do whatever you like in the current generator context
  });
};

The generator algorithm would work as follows:

1. Instantiate a new base generator
2. Pass the generator consecutively to all tasks as registered in 1
3. Run prompts
4. Run other procedural methods in the order specified in the registerGenerator method

---

So this may seem like a major push away from how everything is done now, but I think it offers a certain flexibility to both the end user and plugin authors. It might be possible to support legacy generators as well, since the "root generator" that is passed through consecutive functions could be instantiated based on the base generator class or any other legacy generator class.

[SBoudrias]

@rvangundy About your first though, don't you feel the base idea is resolved with the use of .yo-rc.json file?

[rvangundy]

@SBoudrias Yes, that does solve the configuration issue.

The second part to the first thought is the ability for the end user to compose their own flavor of generator through the registerGenerator function. Pushing the composition responsibilities up to the end user allows for greater flexibility at the config level, and forces generator module authors to be more generic in their implementations. I think by encouraging composition deeper in generator code you may create a lot of complex dependency chains and more rigidly designed generator modules.

[SBoudrias]

We sure plan on giving end user the choice of the generator being composed. But Yeoman is a scaffolding tools, as such, it shouldn't need to be configured by a config file prior to be runned.

IMO (in regard to my previous point) all the composability features should be managed and available directly in the scaffolding flow bundled in the tool.

[addyosmani]

@csantanapr nice work on the generator! This brings up interesting best practice questions about unit testing composed generators. Each individual generator you call will likely have its own unit tests but I imagine things get complex when they're executed together in sequence. Could you summarize some of the issues you ran into so we can keep these in mind?

[csantanapr]

@addyosmani The issues I hit was calling the generators in sequence and passing options to each one in the chain, I failed trying to do it in unit test.

I want to be able to have a simplified way of specifying all options upfront, and then call a single function that starts the sequence.

var allOptions = {
"gen1": options: {
"opt1":
},
"gen2":options: {
}

rungenerators(["gen1","gen2"],allOptions,function(err,results){

[JohnMorales]

Any thoughts on supporting files that can be modified by a user? It seems like if you were to address files beyond paths then it would be possible, at least for files that have structure.. e.g. /Gruntfile.js#yeoman/initConfig/deployPack, then you could inject code into the file without overwriting the whole file. Therefore persisting changes made by other generators and/or users.
it would of course require parsing the file.

I realize this is a bigger change, but it would satisfy two needs for the price of one.

[SBoudrias]

@JohnMorales We already worked on some tools to allow generators author from injecting content into a file. For example, the whole wiring mixin methods are helpers to edit HTML content.

Then, for more specific use cases, it's the job of the author to use code modification tools like AST Query (for JS) and Cheerio (for HTML). I don't think Yeoman need to wrap these tools because they already offers a façade to simplify file edition.

[JohnMorales]

I see your points and I didn't know these tools existed, I'm having a hard time then seeing why don't we leverage them for this? Are they not good enough for the job?

[SBoudrias]

I personally use them in Generator-BBB. We're going to improve documentation soon, I believe tips and tools to help with some common tasks is in the scope. yeoman/yeoman#1259

[robwierzbowski]

Composability fundamentally changes the way authors will write and users will use generators. If APIs can be simplified and improved by breaking compatibility, I don't think Yeoman should shy away from a 2.0 release, or a composable / classic split.

[robwierzbowski]

The biggest challenge I see is in composing custom tasks in the Gruntfile. The serve and build tasks in projects I've worked on have gotten pretty byzantine.

Enforcing more workflow/directory standardization and breaking generators into atomic pieces with their own conventions (asset preprocessor generators (Sass, Less, CoffeeScript), asset postprocessor generators (autoprefixer), asset minifiers) will probably help.

[eddiemonge]

should this still be open?

[SBoudrias]

@eddiemonge I think it should, we still need to take care eventually of user initiated composition.

[gliwka]

Two years later, any plans to implement this?

[SBoudrias]

@gliwka No immediate plans.

[alexbab]

Hi,
https://github.com/yeoman/generator-node/pull/186/files
I can't get this to work does it mean it hasn't been implemented?

[mischah]

Hi @alexbab,

I would recommend asking support questions over here: https://gitter.im/yeoman/yeoman

[alexkreidler]

See #1084 (comment) for an update on the ecosystem as of 2019

[github-actions]

This issue is stale because it has been open 15 days with no activity. Remove stale label or comment or this will be closed in 5 days