Provide promise based API for core libraries

[olalonde]

I couldn't find any discussion related to that and I've been told core contributors aren't a fan of promises but I thought I'd kick start a discussion anyway. I had assumed this was discussed at length already but couldn't find any relevant links.

The proposal would be to make all async methods of the core library return promises. This could be done in a backwards compatible way by using something like bluebird's nodeify (now asCallback) which would allow both errback / promise styles to co-exist.

[Trott]

Here's one place it's been discussed: nodejs/node#11

[calebmer]

But that issue is closed pending future discussion. So this is future discussion 😊

[Trott]

Right, this is a better place to discuss it. I was providing the link for reference since OP said they couldn't find any previous discussion. There's a previous discussion for context.

[olalonde]

@Trott thanks. Last comment seems to imply it's ok to move discussion here, so will keep open.

[yoshuawuyts]

I think nodejs/node#11 (comment) sums up well why Promises have no place in Node core. Node and LibUV aim to resort to the lowest common denominator and provide the smallest API possible. Adding more ways of doing the same thing that can already be achieved by userland packages is against Node's philosophy.

Last I checked on the Node TC meetings the status was that promise rejection handlers will be added as V8 implements them, making Promises slightly less harmful. In terms of TC members some seem to be excited for potential async/await control flow and some are vocally against Promises as a way of handling async.

I feel that this topic is completely chewed out, and this issue best be closed as I suspect it might (again) spin into a very misinformed back and forth.

[calebmer]

I don't think it's a matter of if promises will be implemented (sorry everyone) I think it's a matter of when. Issues and PRs like this will keep popping up until promises are integrated. That's inevitable. I say keep this issue open as it does provide an outlet (and a URL) for anyone who wants to contribute to the discussion vs. seeing a hundred more issues from now until the promise integration.

[olalonde]

@yoshuawuyts let's imagine for a moment that Node core was initially designed with a promise based API, almost all the same arguments could be made about supporting an errback API. Let's try:

First, promises are not as bad as you think and there are some simple, effective ways to organize callback driven code to avoid common complaints.

Second, callbacks have costs and benefits like anything, and while the benefits are well-advertised, the costs can be hidden and frustrating.

Third, the callback is the fundamental unit of asynchronous programming in JavaScript and Promises are just callbacks with added semantics (a specific type of callback if you will). There is no single abstraction that's perfect for all use cases.

Fourth, promises aren't a silver bulllet. There are whole classes of use cases where promises aren't the right abstraction [...].

Well, I don't have to alter that one. There are some places where promises are indeed not a good abstraction, like event and stream APIs.

Fifth, the design philosophy of node core is to encourage maximum compatibility and discourage decisions that lead to incompatibility. While it is trivial to turn a promise based API into a callback one, the opposite is not always true.

Sixth, integrating with c++ bindings/libuv has only been possible using callbacks. The core team is committed to keeping node lightweight and fast by providing APIs that are as "close to the action" as possible, so they're not going to be introducing a new abstraction to asynchronous programming in core anytime soon: http://blog.trevnorris.com/2014/02/nodejs-es6-and-me.html

In my mind, this is the only substantial argument against promises in Node. From the linked blog post:

When the v8 team introduces an API for generators or promises I will give it honest consideration.

So I guess the question is, has v8 introduced this yet? If not, I'll be happy to shut up until then :)

[Qard]

FWIW, async generators are in the works, which would work pretty nicely for streams. It's like async/await, but you get an await per chunk in a for/of loop. https://github.com/zenparsing/async-iteration/#readme

[jakearchibald]

Returning a promise if a callback/errback is not passed would work.

Promise-based APIs shouldn't thow, they should reject instead https://www.w3.org/2001/tag/doc/promises-guide#always-return-promises - so calls without a callback/errback ideally shouldn't throw. Would this be a compatibility risk?

[caspervonb]

Returning a promise if a callback/errback is not passed would work.

Promise-based APIs shouldn't thow, they should reject instead
https://www.w3.org/2001/tag/doc/promises-guide#always-return-promises - so calls without a
callback/errback ideally shouldn't throw. Would this be a compatibility risk?

Callback vs promises will have different semantics by definition, to keep compatability with error first callbacks, we could rethrow if the callback is present.

function stat(filename, callback) {
  let promise;
  // ...

  try {
    // ...
  } catch (error) {
    if (callback) {
      throw error;
    }

    promise.reject(error);
  }

  // ...
  return promise;
}

[jakearchibald]

Yeah, that's what I meant. The compatibility risk is if someone already does:

whatever.asyncThing();

…where the async throw is meaningful to them, but they're happy to fire & forget the rest. The change we're proposing would change the behaviour here.

[caspervonb]

Ah.. now I'm following you. However omitting the callback from an async function is generally an error, or simply undefined behavior. Documentation does not mark for example, the callbacks in the fs module as optional, they are required.

Taking from the C++ world, if you rely on undefined behavior you're toast. That's my two cents on how this should be treated.

[jakearchibald]

Agreed. Given that, if no callback is provided, return a promise and treat any exceptional failure as rejection, otherwise go with the current behaviour.

[bevacqua]

I like the idea of returning a promise if no callback is provided. I think that'd be pretty backwards compatible.

Also, as an equivalent of event emitters throwing when no error handlers are present, I'd have thrown errors bubble out of the promise if no errback handlers were registered (again, when no callback is passed).

E.g:

// throws at the location of the promise
new Promise(() => { throw 'a' }).then(() => console.log('foo'))

// doesn't throw
new Promise(() => { throw 'a' }).catch(() => {})

Lastly, check out my OP here for extra arguments in favor of promises in Node: nodejs/node#4596 (comment)

[mikeal]

One of the main objections to this has been performance. If someone wants to put together a PR and see how it performs compared to the current API, and quantify any performance cost this would cause on the current API, that would probably be the best way to move this forward.

[jakearchibald]

@mikeal what would 'ok' look like?

The V8 team is starting some promise optimisation work, so performance is only going to improve there. If we can introduce promises with negligible impact to using callbacks, that feels like a ✔️ to me.

If using promises turns out to be significantly slower than callbacks, that's something we can give to the v8 team to fix.

[ChALkeR]

@mikeal Afaik those are still slow, and it's v8 to blame — e.g. Bluebird is very fast. Perhaps this has to be changed on the v8 side, and then the speed argument would be much weaker.

[mikeal]

Let me clarify. If this new promise API is slower than the callback API, I don't think that's a huge deal. The bigger dealbreaker is if there is any performance impact on the existing callback API by adding support for promises. Node.js has a general policy that things do not get slower from release to release but there's no real policy about new APIs being equal in performance to prior APIs.

[billinghamj]

Ultimately, if the user of an API is concerned about performance, they can proceed to use callbacks instead of promises - the existing callback support is not going to be removed because it would break stable APIs.

The biggest problem I see is preventing confliction with APIs which assume synchronousness when a callback is not provided. For example:

sync & async: https://nodejs.org/api/crypto.html#crypto_crypto_randombytes_size_callback

Most APIs I've seen use this convention though, which would be compatible without caveats:

• sync: https://nodejs.org/api/fs.html#fs_fs_readfilesync_file_options
• async: https://nodejs.org/api/fs.html#fs_fs_readfile_file_options_callback

[madbence]

it's just a breaking change... with semver, that's not a very big deal

[billinghamj]

@madbence Node has very strict API stability rules, which I believe persist across major versions because it's hard for package maintainers to check engine versions - https://gist.github.com/isaacs/1776425

[calvinf]

For APIs that would be difficult or impossible to update without breaking compatibility, would it be possible to create new methods that use promises?

[bevacqua]

I don't think introducing promises necessarily has to be for every API endpoint ever. At first, and for a long time, maybe forever, it could just be for the most commonly used APIs, such as fs.readFile.

[billinghamj]

@calvinf We could add new methods like crypto.randomBytesAsync, but that's not super ideal in terms of keeping a consistent API. Probably a worthwhile tradeoff though.

[caspervonb]

It's primarily going to be for fs, dns, child_process.exec, etc. Events and streams are not thenable patterns, no one is proposing doing that right?

[benjamingr]

One of the main objections to this has been performance.

It's worth mentioning that v8 exposes internal hooks for promisification.

There are two conflicting time constraints here:

• V8 promises are still pretty slow, at least much slower than userland solutions.
• When async/await lands (it already landed in Chakra, for example) our callback APIs would become mostly irrelevant anyway.

(Please, it has been shown time after time that "promises are slow" or "callbacks are closer to the metal" is FUD, multiple languages use promises as their core abstraction for asynchronous actions).

[yoshuawuyts]

our callback APIs would become mostly irrelevant anyway

Be careful with statements like that, as a considerate amount of people (including me) prefer callbacks over coroutine style declarations. I'm suspecting it's a matter of taste, and blanket statements like that can easily polarize discussions. I think it's best to keep this discussion pointing towards how promises could be implemented into the Node API and what the results of that would be.

[mikeal]

Be careful with statements like that, as a considerate amount of people (including me) prefer callbacks over coroutine style declarations.

Seconded. Also, this is more than a matter of preference, there's thousands of modules in npm that rely on the existing callback API. I don't see a future where we can ship a version of Node.js where all of them break.

[billinghamj]

The existing callback APIs are clearly not going to be removed for the foreseeable future. Please stop raising concerns around this - it simply isn't going to happen.

[yoshuawuyts]

Yo, can we lock this thread? It's been closed for a while and over 30 people are now being spammed for something months after discussion had ended. Thanks!

edit: just realized this is the second time I'm calling for this. cc/ @jasnell think it's time

[benjamingr]

I'm fine with locking this if we provide users with an alternative place to ask their questions. Giving people a locked thread and nowhere to turn is not where I think we want to be in terms of community openness.

[olalonde]

@benjamingr don't know if this has been considered but one way to mitigate forward compatibility issues with older code bases, would be to bundle the introduction of potentially breaking changes in the promise based APIs with import/export module loading (e.g. const fs = require('fs') would return a completely backward compatible API while import fs from 'fs' could return a promise based API which might otherwise potentially break in older code bases).

[benjamingr]

@olalonde yes, it has been considered several times as well as several other alternatives. Thanks for the suggestion though. See nodejs/node#5020 if you're interested in the previous discussion.

[iamstarkov]

@yoshuawuyts

discussion had ended

can i ask you to point to any discussion's result?

[benjamingr]

@iamstarkov pretty much #25 (comment) - it's a lot of work, very smart and diligent people have tried and so far their efforts are stagnating (see nodejs/node#5020)

You can also support nodejs/CTC#12

[jmm]

@yoshuawuyts Can I ask why it would need to be locked -- can't people just unsubscribe if they don't want notifications?

[pleerock]

even async/await landed to node and its still not supported? Noone using callbacks these days, everybody are using promises.

[styfle]

If I had to guesstimate - it'll get postponed until V8 JS gets C# like async stack traces which would solve the postmortem concerns some people have.

@benjamingr It appears that you can debug async stack traces using the inspector protocol. See the blog entry for VS Code version 1.11 for more info.

Is this what you were waiting for?

[benjamingr]

@styfle real async stack traces are very different from what you have in V8 at the moment, you can't - for example - go to an earlier frame and inspect its local stack variables - you can just see where it came from.

[olalonde]

@benjamingr you can't - for example - go to an earlier frame and inspect its local stack variables I don't understand... is this currently possible with non promise code?

[benjamingr]

No, but non-promise Node throws synchronously on errors where promise node waits for the next micro-task.

[damianobarbati]

@pleerock totally agree. Are we going to have 10 years of "callback style" Node core API while the world and ecosystem is fully async/promise based? :(

[addaleax]

Fwi, here are some current relevant discussions that are going on:

• domain: support promises node#12489: PR: domain: support promises
• util: add util.promisify() node#12442: PR: util: add util.promisify()
• Support awaiting values through util.awaitable CTC#12: More general discussion around implementing a feature like util.promisify()
• Post Mortem Analysis with Async APIs CTC#114: Discussion thread for how Promises interact with post-mortem debugging

Are we going to have 10 years of "callback style" Node core API while the world and ecosystem is fully async/promise based? :(

No.

[kaven276]

to keep existing callback based API act as before, but add promisify support without any extra import/syntax, maybe this is good.

const fs = require('fs');
fs.readFile('file', (err, fileConent) => {} );  // for callback based API
fs.readFile('file', 0).then(...).catch(...);  // for promise based API

if the original callback parameter is not a function, but a false value, it mean a promise will be returned.

advantages

• API for promisify support is simple
• callback code and promise code is similar
• if you are used to callback, switch to use promise is very simple
• 0 can be used as false, so code is short for return promise
• existing code will not broken

if a callback-base API's last parameter before callback function is optional, it should be a option object, an object is a true value. @billinghamj

or the return-promise flag in replace of callback function can be a special value or an special global object ( maybe is global.Promise )
fs.readFile('file', Promise) will return Promise cause the last parameter is Promise itself.
API implementation will check the last augument if === Promise; @billinghamj

[billinghamj]

@kaven276 I imagine that could conflict with various APIs which have optional parameters. Booleans and/or numbers may already have another meaning if put in the place of a callback param. There are a very large number of core APIs and their interfaces are very varied, so the solution must be guaranteed to work in all cases.

[styfle]

nodejs/node#12442 landed 2 weeks ago. It's available in the nightly builds and should be available in the final Node 8.0.0 release. I wrote about it on medium.

This gets us really close to having Promises in core and I suggest everyone play around with it now if you haven't already to see how it may (or may not) solve your problem.

[billinghamj]

Hmm it is a good step, but it's just so little and so late. If it allowed e.g. require('util').promisify(require('fs')) then it'd be useful

[kaizhu256]

-1
adding promises to node-core is a terrible idea. node-core should have stable design-patterns. adding promises will lead to contributors needlessly refactoring node-core with promise design-patterns (like what happened with let and const).

and then what next? are we going to add generators and async/await design-patterns as well? i don't want node-core's api and design-practices to turn into chaos, like what's currently going on in frontend-development world since es6/babel was introduced to it.

[benjamingr]

@kaizhu256 what's unstable about promises? According to surveys the vast majority of Node.js users already use them anyway - and at the moment there is a hassle involved in order to use the language.

[kaizhu256]

@benjamingr, you are then encouraging people to try and promisify sockets, just because they can (and request and response objects). can you imagine the never-ending code-refactoring/debugging this will cause and tie up resources to actually shipping a product?

[kaizhu256]

i would say joyent's stable stewardship of nodejs was a good thing, which allowed people to build incredible things, without worrying about node's api constantly changing. you guys risk turning nodejs into a joke (like what's going on with npm after they mucked up npm-install and npm-publish).

[madbence]

afaik backward-compatibility is taken very seriously in node-core, that's why util.promisify was introduced in node@8 instead of changing the existing api. promises are already part of the core.

[pleerock]

node-core should have stable design-patterns.

yes, but stable isn't the only criteria about node-core's design patterns. They also must be a good design-patterns. But we all know that "stable design-patterns" you are talking about are callbacks which produce a callback hell and unmaintainable code, we all know about. This means that such "stable design-patterns" are actually anti-patterns. Does node-core need stable design-anti-patterns?

[benjamingr]

@kaizhu256 promisifying sockets would never happen since promises are for one time things. The goal is to provide people with the most convenient API. If you look at the prior art - absolutely no one is suggesting breaking APIs.

I do however have every intent to pursue async iterator support in core once those land in V8 and we have already been doing work for it.

Note that stability isn't being sacrificed here.

[benjamingr]

@pleerock let's please not turn this into a "promises vs. callbacks" debate or discussion. I am interested in engaging and discussing with @kaizhu256 because his point of view is important to me and I am happy they chose to engage with us.

I do not want to belittle their experience or use case or to assume mine is more valid than theirs. I would like to convey that Node.js is committed to API stability and to discuss how they feel adding support for promises might impact that.

Thanks :)

[kaizhu256-heroku1]

@benjamingr i liked joyent's vision that all builtin modules should work towards reaching eventual frozen status, for the sake of api stability. people like me would like nothing better than to see fs, net, etc. eventually get to frozen status. i would be against anyone deciding to revisit existing builtins to tack on promise or generator apis to them (there's fear these things will end up rewriting the entire module and break stability). i'm ok with people creating new builtins like fs2 or fsPromise, but leave the existing modules alone.