Provide a combined ampersand module

[remko]

I didn't know where to post this (which is part of the point I'm trying to make here), so I just posted it in what I think is the most important Ampersand module.

One of Ampersand's primary features is that it is a collection of modules that can be used independently, and that there is no big module necessary. While this is a good goal design-wise, it poses some practical problems for Ampersand users like me:

• My npm-shrinkwrap.json is currently 2300 lines long, 2000 of which are Ampersand.
  As someone who likes to be on top of all the dependencies of my production project,
  this scares me a lot. This is also not what one would expect from a 'simple, no-magic'
  framework. (Backbone.js only has underscore as a dependency).
• I seem to recall only upgrading specific ampersand modules once or twice, and yet
  the dependencies of all Ampersand modules combined are all over the place: some modules
  depend on underscore, some on lodash.*, and I also see some other dependencies
  that could (and might have been already) be replaced by lodash.*.
• Whenever I detect I need to upgrade a module, I'm too scared to do it these days.
  Especially if it's a module that has some dependencies or dependents of other
  ampersand modules, I'm worried about what will break because of incompatibility,
  what other dependencies I'll end up pulling in, etc. For example, I tried upgrading
  a module last week, but something somewhere else in Ampersand broke. I don't know
  if it is actually a bug, or my own fault of incorrectly updating another module.
• It's hard for me to keep track of what changed to which ampersand module; every
  module is upgraded independently, no changelog is kept, so i would have to
  go through all the git commit logs of all modules I indirectly depend on.
  This means I only update when I spent some time discovering a bug myself,
  which is too late.
• An extra problem in commercial settings is that each module's dependencies (and subdependencies) need to have their license checked by legal teams. The size of the dependency tree in itself means this is a significant task, but this also makes it a hassle to update a single ampersand dependency, as it needs careful checking what the upgrade did, and whether it didn't cascade other dependencies.

This is why I would like to have a combined ampersand module with all (or at least the most important ones with common dependencies) ampersand modules combined. Advantages:

• This greatly simplifies the dependency tree in a project using ampersand, and
  simplifies mental tracking of the ampersand dependency.
• Dependencies of Ampersand are maintained and updated in sync, so it's easier
  to track those as well
• Upgrading ampersand feels much safer, as I know that anything breaking will
  be a problem with code, not some strange interdependency problem or something
  I did wrong.
• Keeping track of what changes in ampersand is easier, because I just have to
  look at the changelog / commit log / diff of a single project.
• It makes it easier to report global ampersand issues such as this one, as there
  is an all-encompassing issue tracker
• Probably not important, but combining usage may give a more accurate score on
  npm.js.
• I'm not an Ampersand developer, but I can only assume that this would make
  some aspects of developing modules (especially upstream ones) and
  figuring out bugreports easier.

I'm not sure what the easiest practical way to get to such a module is. Maybe a first step could be to hand-pick a combination of modules+versions that play well together and have similar dependencies, and use a script that combines all these modules into a single module that is maintained separately. This would solve many of my problems. However, I think ultimately it would be simpler to have a lodash-like setup, where a single module is chopped up into smaller modules and published for those who like that sort of thing.

An extra optimization (which would simplify the dependency tree even more) is to depend on lodash as a whole instead of dozens of small lodash parts which in turn depend on several other lodash modules. If size of unused dependencies is a concern, tree-shaking could get rid of the unused lodash parts in the future anyway?

[cdaringe]

hey @remko,

I don't think the team has this on the roadmap, but would certainly welcome a community module. It could be as simple as:

// ampersand-mega/all/spine/agg.js
module.exports = { State: require('ampersand-state'), Model: ..., Router: ... }  // eg ampersand-backbone ;)

Were such a module to be released, the internals & API of the aggregated package would change, and consuming APIs would still need to adapt accordingly. What you gain is the ability to flip back-and-forth between multiple &-js modules' versions quickly. This all-and-all would be a fine thing! IMHO, however, the problems you noted don't really get addressed by a singleton package. The legal teams will still need to sniff nested licenses. The UI tests would still need a full sweep.

I humbly believe that the best we can do as a framework team is (a) keep on top of denoting clear changelogs and (b) keep on top of writing many, diligent tests. You've noted that changelogs aren't kept. This isn't fully true, but hasn't been rigorously followed either. This is definitely a place for improvement. cc @AmpersandJS/core-team for making a changelog process "more official."

I've written and deleted various responses to this, unable to select the right wording. However, to summarize simply, as a fellow user, I don't believe that further simplicity or clarity is achieved by aggregation.

Some specific responses to some of your bullets:

My npm-shrinkwrap.json is currently 2300 lines long, ...

a big project is a big project. I think having "many dependencies" is unfortunate in that it generates a long list, but it's the beauty of the unix philosophy. I personally wouldn't claim that "many dependencies" is a strong complexity adder.

... modules combined are all over the place: some modules depend on underscore, some on lodash.*...

I hope this is not the case. the group made a big effort to lodashify almost everything. at least the official modules

Whenever I detect I need to upgrade a module, I'm too scared to do it these days

Yea, better changelogs could definitely help--I'm with you here for sure. However, that's what semver and testing are for too, which are invalueable assets in any huge project. sounds like yours is a pretty big one!

This greatly simplifies the dependency tree in a project using ampersand ...

At a cost. A cost of ambiguity of purpose, and refactor-ability of the consuming code. ¯\_(balance)_/¯.

Keeping track of what changes in ampersand is easier ...

this is a complicated claim. for some us, perhaps yes. for my usual use cases (picking and choosing a small subset), I would not think so.

It makes it easier to report global ampersand issues such as this one...

I'm not confident such a thing exists ("global ampersand issues"). I think what you are getting at is when some change or bug affects many modules? As a counter, wouldn't it feel incorrect to post the issue in a global place versus the offending package/repo?

Anyway, that's my long response to a long question. :) I respect your opinion as a peer and active ampersander, so I hope nothing came off brash!

[remko]

Hi @cdaringe, thanks for taking the time to respond.

It could be as simple as:

As you indicated, although this could serve as an initial step of an amalgamated module, this unfortunately doesn't solve any of the problems I have.

I humbly believe that the best we can do as a framework team is (a) keep on top of denoting clear changelogs and (b) keep on top of writing many, diligent tests

Keeping decent ChangeLogs only marginally helps me: To upgrade a single module, i would potentially have to go through the changelog of all dependent and depending modules to see what else changed, after figuring out which versions I need to look at in these changelogs by analyzing the package.json of each of these modules (because the versions are completely independent). This is unfeasible, even for a small project.

Also, I'm not contesting the quality of the modules individually, so test-wise, things are fine.

a big project is a big project.

That's the thing, I don't consider it a big project. I just happen to depend on a few (not even that many) ampersand modules, which blows up everything. If I would have created TodoMVC, the numbers would be the same (except even more Ampersand share in total).

[lodash] I hope this is not the case. the group made a big effort to lodashify almost everything. at least the official modules

I checked, and they did. But because of the 'everything is a module', I am able to add and/or upgrade single modules when needed (that's the advertised feature), and so I'm depending on different modules at different points in time (points where underscore was still a dependency). So as a library user, I'm again left with the mess. If I wanted to get rid of this, I'd have to upgrade every module i'm depending on, and this is way too much work and risk.

(On an aside, I can imagine something like an ampersand-global move to lodash would probably have cost lest effort in orchestrating and propagating dependency version changes across the different ampersand modules if ampersand would have been a single module, but we'll never know)

At a cost. A cost of ambiguity of purpose, and refactor-ability of the consuming code. ¯_(balance)_/¯.

I don't get this yet. I don't see how importing different modules is any different from importing different exports from a single module.
The only thing I see happening is that:

• the major version of ampersand would bump quicker, and that you'd have to do a major semver dependency update even if you don't rely on a major change. For a project that is not in infant stages anymore, this is pretty normal, and I don't see why this would be a bad thing. It just makes it easier for me to decide whether I want to do a major semver upgrade of my ampersand dependency or not, because I just have to skim through a single changelog at the items marked with 'breaking'.
• I can no longer use an ampersand-view which depends on ampersand-state v4.x, and depend on ampersand-state v3.x at the same time. That sounds like a good thing to me.
• I can no longer upgrade my dependency of an upstream module if a downstream module had a major version change and I don't want to change my code. If this is a problem, Ampersand will need to keep stable and development branches. This is an actual disadvantage, but one I'm much more happy to live with than the disadvantages of the current approach, because it also puts the disadvantage with the library developer (because of the branch management) than the user, as it should be.

As I said, if you're really crazy about different modules for some reason (i don't see a good reason, especially if you have tree shaking, unless perhaps the last issue), nothing prevents Ampersand from splitting up this single module into small published modules, like what Lodash does. It's just much easier in this direction than the other way round.

I'm not confident such a thing exists ("global ampersand issues").

Look at the Ampersand.js roadmap: many items are global ampersand issues. But this isn't really an important discussion point for me.

so I hope nothing came off brash!

It didn't.

In the meantime, I created a quick-and-dirty proof-of-concept of an amalgamated ampersand module, and depending on this in my project brought my npm-shrinkwrap.json down from 2300 (2000 of which due to ampersand) down to about 400 lines, only 120 of which are ampersand and its dependencies. And this isn't even including that many ampersand modules; a project including more of the views etc. would have even bigger reductions.

[codepunkt]

My 0.02$...

Keeping decent ChangeLogs only marginally helps me: To upgrade a single module, i would potentially have to go through the changelog of all dependent and depending modules to see what else changed, after figuring out which versions I need to look at in these changelogs by analyzing the package.json of each of these modules (because the versions are completely independent). This is unfeasible, even for a small project.

In our project, automated and manual testing in combination with thorough error-logging ensures that errors after the upgrade of a module are found early and noticed immediately in case they reach production. So this is usually not too big of a deal.

That's the thing, I don't consider it a big project. I just happen to depend on a few (not even that many) ampersand modules, which blows up everything. If I would have created TodoMVC, the numbers would be the same (except even more Ampersand share in total).

It probably isn't. Our shrinkwrap.json is at ~10k lines and i wouldn't consider the project big. It's just a real single page application and not a basic example shown in some kind of readme or documentation.

Personally, i like being able to think, reason and discuss about small, isolated parts of the framework. If they were combined tasks like searching the issues would be much more time-consuming. But i guess that's just a matter of taste and both options have their pros and cons.

[remko]

I just want to mention that upgrades like the one in #230 sound like they would be a lot less painful for both AmpersandJS maintainers and AmpersandJS users (again, I need to figure out module by module which version i need to upgrade to in my package.json to be sure I don't have clashing lodashes).

This 'small modules' approach is becoming a maintenance liability for us (a feeling amplified by the pad-left saga), and I'm not convinced I'm the only one who's put off for using AmpersandJS because of this.

[cdaringe]

@remko, ya, i'm opening up to the idea. needing to review other teams' monorepo formal approaches some.

[cdaringe]

yep, now i'm just outright in full support. keep bundle sizes lower, improve cross-dep deployments. i was short-sighted before, apologies. the recent npm unpublish fiasco, &-state upgrade version debacle, and lodash v3 => v4 bumps are all advocating examples.

@AmpersandJS/core-team, something to consider. long term, it would be nice to migrate to a lerna-like approach for version control & deployment.