-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
async modularized #984
async modularized #984
Conversation
This reverts commit 112a382.
Hi! I'm pinging in at the request of @Kikobeats. First things first, wow this is an awesome amount of work you've done 👏! I total dig modularization 👍, though I prefer the modularized approach of something like var series = require('async/series'); because it simplifies things. You don't have to worry about managing issues of or syncing across all repos for simple scaffolding changes in linting, style, or editor configs. I've had folks express concern with the load cost, in time, of synchronously loading 100+ modules for the monolithic build. The cost is then multiplied by the number of different versions of your package, for me lodash, in anothers dependency tree. Which is why I opt for a pre-bundled monolithic build for the default kitchen sink. var async = require('async'); // loads 1 file The modularized single package also favors bundlers like webpack, browserify, & rollup which are gaining in use. This is because they don't have to worry about duplication. With individual packages I've had developers push back on things being too modular. Devs tend to tolerate only a few package dependencies (the fewer the better). Which means for the individual packages a bit of inlining is required to keep the dependency count low. However, duplication leads to larger builds. Also, when reporting the size of Async look to its gzipped minified size which is only 4.26 kB. I would also dampen the idea that offering a build tool will solve the problem. In my experience folks don't want to build something. They want something that already works for their package manger of choice. Which means pre-built and packaged versions of things. |
Got to agree with @jdalton on the approach. @kikobeats- tremendous work man! Lodash and async are the two most important libraries in the node ecosystem right now (IMO) and the idea of seeing them modular used in a similar fashion is really awesome. |
First off, this is an impressive amount of work! Its a bit different that what we were planning, but then I also realize that the latest plan only exists in a chat between me and @megawac and in my head. We also talked about this a little in #914 . There are many goals for async in regards to modularization.
Since our earlier discussion in #720 / #748 , two things have changed: The ES6 spec and its modules were finalized, and @Rich-Harris has put a lot of work in to rollup, an ES6 bundler that supports tree-shaking (removing unused imports). Here is the plan I've been envisioning: We author async as a set of small ES6 modules, one per method. This is similar do what you've done, except it would use ES6 imports rather than To support #1, we would have a To support #2, we take the CJS output, and To support #3, we'd take advantage of lodash-es. We can import just the methods we need, or To support #4, we do two things. First, we individually Rollup each module associated with a public method, and place them in appropriate folders as CJS modules. For example, For #5, we can break up the readme, either now or later. Each public module would have a large JSDoc block with the signature and prose-form docs, and we can work out the doc publishing strategy later. Regarding #6, initially we were thinking of using browserify, but the ES6 module strategy will prove to be much more lightweight. Async uses its own methods a lot internally, and there are several helper functions that are used in many places. If each module has to re-import these as CJS, then there are a lot of Regarding #7, im a fan of keeping everything in a single repository. I don't thin we need an org yet. Since async uses it's own methods internally, and we occasionally make large sweeping changes to how things are implemented, it's much easier to keep things together. I also think people would rather Overall, I think we are going to see the ES6 module + tree-shaking strategy being used more and more, and I'd like to see async support it. It's a nice compromise between small modules and discoverability/convenience. |
Just to be upfront – right now, if you
This would work, but you would end up in a situation where someone who required, say,
Rollup does allow you to import from CommonJS modules, using https://github.com/rollup/rollup-plugin-npm and https://github.com/rollup/rollup-plugin-commonjs – this might allow you to generate a UMD build directly with Rollup and forgo the Browserify step. |
Was a good idea create the PR instead of continuing coding behind the dark, now I see more problems associated that my mind avoid 👍. I associate a package per each util/async method because I thought it would be easier to maintain: If N modules depend of the X package and X have a bug, releasing a new version X and using the last X version fix the problem. But the point is that I'm fixing a problem creating another one: is it necessary have too much repositories for simple code? Create a request to get noop implementation is a little useless (in time terms and CPU por resolve the dependency reference) At the beginning I was worried about the quantity of time since install async dependencies (divided in modules) to be ready to code. I testing and need around 15s and I was that is ok. But now I see another better point: Instead of fragment the library into repositories, we can fragment the function into files. Instead of create a file and export the method, like: 'use strict';
module.exports = function noop () {}; Just define the method:
Doing this per each function and having a map of functions references that each method need (for async.util and async) we can inject the function in the build, like copy/paste the function into the build file. Just need to do is control the functions to inject to avoid duplicate code and use the same variables name to keep the naming reference. I think that this is the approach following by lodash-cli because I see that each module have the same code that the full library (including documentation!) Also I think that this new approach resolve the problem to load the library using ES6 modules interface |
I just want to prefice with great start and thanks for attempting to do this. As @aearly mentioned there has been quite a bit of planning on how this should be done (I'll elab when I'm off my phone). anyway, the most significant amount of work here is developing a build tool to do some grunt work for rollup to minimize compiled asyncjs code. A publish script for npm should also be configured which would enable users to do require(async/each). Managing custom npm repos would be a possibility as well. Overall, the build yool would be responsible for significantly less than the scope of lodashs cli and probably remain internal I started doing some of this in October but have been very busy in school and work. If you want to continue the PR I'll message you on gitter about some specific changes. I will also have some time to assist/lead on this over the next week or 2. The most significant change off the bat is that all the code should live in this repo, npm modules can be made on build (possibly). I'm a bit torn on how documentation should be done. Splitting out readme into modular folders is probably alright but I'd prefer jsdoc generated documentation. Take a look at ramda/ramda repo, which has src file organization similar to what I was proposing. |
My WIP is occurring on this branch(https://github.com/caolan/async/tree/modularization) - an example of the bundles are here https://github.com/caolan/async/tree/modularization/build. Still need to develop scripts to publish the bundles to npm/git tags. |
Looks good @megawac . I'm really happy with how the main bundle looks. 😮 One thing I noticed -- the new I also noticed that the modularized methods use |
With webpack and its webpack.NormalModuleReplacementPlugin you can swap modules like plugins: [
['lodash/internal/baseIteratee.js', './toFunction.js'],
['lodash/object/keys.js', '../internal/baseKeys.js'],
['lodash/object/keysIn.js', '../internal/baseKeysIn.js']
].map(function(pair) {
var resourceRegExp = RegExp('\\b' + _.escapeRegExp(pair[0]) + '$');
return new webpack.NormalModuleReplacementPlugin(resourceRegExp, pair[1]);
}).concat(
new webpack.optimize.OccurenceOrderPlugin()
) In the case of Redux the big win was replacing The modularized bundle is 7.36 kB vs. its current 4.27 kB so that's a 3.09 kB diff to work on. It looks like a big chunk is coming from lodash v4's |
Yeah, Some other things small things which can be avoided are I'll open a WIP pull request to move discussion to the relevant issue. |
Thanks @Kikobeats, this was a helpful stepping stone and quite useful for begining the other pr; we're going to close now in favour #996. |
Yeah, thanks for all you've done @Kikobeats ! If you want to help out more, you can submit PRs to the modularization branch. |
Async modularized
What we are
If you check the size of the library (36KB) maybe you think that modularized all methods to works serapately is not necessary. This is because you are think that the only benefit is modualize something is get a less build size, but actually is more a question related with flexibility and have options!.
Async is very useful for traditional node backends. It's use callback pattern, but nowadays promises is in the roadmap of whatever node programmer as more simple pattern.
Should Async support promises and callback by default?
I'm sure that then the size of the library would be more higher than 36KB.
Again flexibility is the question!
Ideally, yes, make possible use both pattern could be nice, but in the real life I didnt find scenario that mixin both patterns.
On the other hand, is very popular packet your node bouild using browserify or something similar... and in this key the size is important. Yes, you think that async is not a browser library, but think about WebWorkers and you will unleash the beast.
How others libraries resolve the problem
If you check others libraries highly modularized (for example, lodash) you can see the value of follow this approach:
How I try to resolve the problem
Internally async use a set of method (currently declared in the main file) that are used for async main function.
Basically more low levels method make more high level methods, so just we want to do is modularize from low to high level.
I created async-js organization and I forked async 1.5.0 code and I started modularizing with this approach:
The new scenario
If you check the new main file basically reference all async.* repositories and this repositories references the low level methods.
This is in the case that you need all async methods, but in other case you can require the methods that you need like:
The only difference is that async main file have the backward compatibility function names, as
async.forEach === async.each
.If you need async.each in your frontend but no more methods, instead of 36KB you have a build of 4KB.
Stuff for the future
It exists a set of things to do to create a more solid ecosystem:
What do you think? :-)
Fix #720
Related #748