util: add util.callbackify()

[refack]

This is a first draft of an implementation of util.callbackify(function).
There is at least one issue that need ironing out: if and how to project the return value into multiple args...

Implementation inspired by @addaleax's #12442
Background discussion at: nodejs/CTC#109

Open discussion

Should we allow customization of the wrapping?

1. specifying a transform for the awaitable before the connection to the callback
2. projection of the returned value
3. tiying in the callback into the promise's life-cycle

---

Original commit descriptions

Add util.callbackify(function) for creating callback style functions

from functions returning a Promise/awaitable

Fixes: nodejs/CTC#109

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

Affected core subsystem(s)

util

[refack]

For how to treat the returned value I choose to:

1. By default pass on as-is
2. Allow the user to ask us to splat the value
3. Allow the user to specify a list of keys that we will project out of the value
4. [Just thought of] allow the user to provide a projection lambda

P.S. treat this as pseudocode, since by the time my Windows machine finished compiling I'll be fast asleep, so I haven't verified it works :(

[refack]

/cc @chrisdickinson @benjamingr @Fishrock123 @nodejs/ctc

[not-an-aardvark]

For how to treat the returned value I choose to:

1. By default pass on as-is
2. Allow the user to ask us to splat the value
3. Allow the user to specify a list of keys that we will project out of the value
4. [Just thought of] allow the user to provide a projection lambda

I think option 1 is definitely best. The other options can easily be simulated in userland by using destructuring in the callback, so I don't think it's necessary to have special behavior to for spreading arrays/objects in core.

[TimothyGu]

I'd protect this with if (fn[kCustomCallbackifyiedSymbol] !== fn) as Object.defineProperty may be quite slow.

[TimothyGu]

I'd fill in undefined for properties that are not present.

[refack]

I was wondering about that too.
Can we find some precedents? lodash maybe?

[TimothyGu]

We should not make toSplat or project public for now, just like for promisify.

[refack]

🤔 not sure, but keeping in line with promisify

[TimothyGu]

-kCustomCallbackifyiedSymbol
+kCustomCallbackifiedSymbol 

?

[vkurchatkin]

This seems unnecessarily complicated for such a simple feature. Is there an example where you would need all that customization?

[not-an-aardvark]

If we keep this property-picking option, then I think this error should be sent with cb(new TypeError(...)) rather than throwing and rejecting the Promise. If someone is using callbackify, then they're probably going to be listening for errors in the callback rather than listening for Promise rejections. (However, I don't think we should keep the property-picking option -- see #12712 (comment).)

[refack]

Yep. Didn't think of that.

[not-an-aardvark]

This could cause confusing behavior if the original Promise rejects with a non-error:

function promiseFunction() {
  return Promise.reject(null);
}

util.callbackify(promiseFunction)((err, result) => {
  // err will be null, which could be confusing because the promise actually rejected.
});

Bluebird seems to handle this by wrapping the rejection reason in an Error if it's falsy.

[refack]

Maybe even if !err instanceof Error?

[not-an-aardvark]

I don't think that would be a good idea because people sometimes intentionally reject Promises with objects. If we wrapped the objects in errors then we would lose information about the properties.

[refack]

Adapted BlueBird snippet

[vkurchatkin]

What if cb throws? We probably want to avoid getting rejected promises from callbackified functions

[refack]

What can we do? who can we tell?
Maybe process.emit('uncaughtException') or 'unhandledRejection'?

[vkurchatkin]

Maybe wrap the whole thing in process.nextTick?

[refack]

So it'll go out of the Promise timeline?
I'd rather explicitly explode a la https://github.com/nodejs/node/pull/12712/files#diff-3deb3f32958bb937ae05c6f3e4abbdf5R1121

[vkurchatkin]

Not a good idea, it's better to allow the error to bubble. Otherwise you'll have to duplicate a lot of logic here for no good reason.

[refack]

I'll check it out... (don't want to duplicate!)

[not-an-aardvark]

Is there a use case for having util.callbackify.custom? util.promisify.custom is useful because some functions have different callback conventions and so the default implementation wouldn't work, but that isn't the case with util.callbackify because all promise-returning functions just return a promise without separate conventions.

[refack]

Is there a use case for having util.callbackify.custom?

🤔 I thought of it as an 'escape hatch' to allow users to do whatever. Maybe adapt to old Promise implementations like our mpromise that doesn't have a catch

I have in mind the use case where this is used by dev A to adapt dev B's and dev C's APIs into a new uniform API, and there is a need to tweak each adaptation a little. So I'm not even sure about hiding callbackify.toSplat && callbackify.project. I think it makes the implementation more complete...

[not-an-aardvark]

I think having customization here is unnecessary given that it's trivial for users to customize the behavior in userland. Adding extra configurability where it's not needed seems like it will make the API bloated.

[refack]

P.S. do you think we should return the Promise
return orig.call(this, ...args).then((ret) => {
https://github.com/nodejs/node/pull/12712/files#diff-3deb3f32958bb937ae05c6f3e4abbdf5R1096
So the user could add a .catch or chain something?

[benjamingr]

Thanks for the work! awaiting changes.

[benjamingr]

I don't think we need these properties - they complicate the PR and with destructuring syntax and a uniform convention none of the benefits it would bring to util.promisify apply here IMO.

For example, it is required for fs.exists to work in promisify but there is no equivalent here.

[refack]

Came to same conclusion. Gone in last commit.

[benjamingr]

I don't think we should support this - with promisify we don't really have a choice for things like fs.exists but here destructuring syntax should be sufficient.

[benjamingr]

Using ...args args.pop and call with ...args is expensive here.

I would prefer to see arguments used, apply instead of call, and an inline slice instead of .pop for performance like bluebird does.

[refack]

Since this will probably go on top of TF+I, is this still a major issue?

[jasnell]

Just noting to keep it in mind, it's not yet clear when we'll be enabling TF+I. It's entirely possible that this could land before that but hard to say for sure.

[refack]

I'll cook up some benchmarks (also look for DEOPTS)...

[benjamingr]

If you run this on top of TF+I (by checking out the appropriate branch) and the results show that these optimizations are meaningless - then we can drop them. I suspect they still affect execution speed on TF+I.

[addaleax]

@benjamingr Fwiw, there have been PRs like #12183 opened by V8 people. I’m good with using rest parameters here, at least for this PR.

[benjamingr]

Interesting, @hashseed care to weigh in?

[benjamingr]

similar comment about not needing a transformer.

[hashseed]

Rest parameters are at least as fast as arguments object, and avoids many performance cliffs. Spread is just as fast as Function.prototype.apply. If I remember correctly this applies to both Crankshaft and Turbofan.

@psmarshall and @bmeurer have more details though.

[hashseed]

In any case, the runtime of this function is likely to be dominated by the chaining of the promise that follows. I wouldn't micro-optimize this just yet, and opt for the cleaner syntax.

[benjamingr]

You have to call cb inside a process.nextTick in case it throws - you can't just abort the process or emit an event since it has to work with domains.

This is what bluebird and other libraries do with their implementations and it's the most reasonable thing to do.

I would also avoid all the if/else here and the projection - which users can just do in a second then.

[benjamingr]

At bluebird, this improves debugging experience. In Node this is not something we can afford - converting non-errors into errors is useful for a library but it's not something the platform should do.

[refack]

Ref: #12712 (comment)

[benjamingr]

I saw that, but we can't assume that if someone threw a falsey value it should be converted to an error. Bluebird does this to overcome the lack of stack trace but node should not make that decision for users. As a platform we should provide a more careful API.

[not-an-aardvark]

To clarify, I didn't suggest doing this so that errors would have a stack trace -- I suggested it because null is a special error value in callbacks which means "no error occurred". If we don't error-wrap, there is no way for a callback consumer to distinguish between "the promise rejected with null" and "the promise fulfilled with undefined".

[benjamingr]

Ditto, this catch is incorrect, and the error must be emitted in nextTick, otherwise this breaks post-mortem debugging.

[refack]

I'm looking into this (and the nextTick)
Ref: #12712 (comment)
P.S. .catch happens on nextTick

[benjamingr]

@refack the problem isn't that catch happens on nextTick, it's that if you abort on unhandled exception the error would be thrown properly and the heap would not be possibly corrupted.

(Also, catch happens after nextTick, not during)

[refack]

So you want an "explosion", callback throwing style? Makes sense.

[refack]

Last commit goes full-on with the options.
For your consideration...

[refack]

Should we return the promise?

[benjamingr]

No, we should return undefined

[refack]

Reason?

[benjamingr]

It gives the user a false sense that the promise is actually somehow related to the callback's execution and creates a confusing API in which the user might expect the callback throwing to reject the promise.

This is why we switched from returning the promise to returning undefined in bluebird.

[refack]

I'm going to document that

[benjamingr]

Basically, people were doing things like the following with BB:

await promise.asCallback((err, data) => {
  console.log("2");
});
console.log("1");

And expected 2, 1 to log (and got 1,2), not to mention people thinking that if they nest callbacks it will somehow magically wait for them.

[refack]

Documented.
But added an option to opt-in to such behaviour 😉

[not-an-aardvark]

Last commit goes full-on with the options.
For your consideration...

I don't think this is necessary either, for the same reasons as #12712 (comment). I think we should just remove these options and the related functionality.

[sam-github]

No docs.... no comment (how could I if I don't know what its supposed to do?).

No node API returns promises, does it? What is the API for? (see above.... "no docs").

[addaleax]

last argument?

[refack]

Fixed.

[addaleax]

Is there any reason for creating objects here? Just passing them as 2 or 3 extra arguments to nextTick should be fine

[refack]

I had something else in mind, but now you're obviously right. Fixed.
(p.then((ret) =>{ret, options, cb}).then(onFulfilled))

[refack]

While the options is still in debate, I added option.catch allowing the user to tie-in the callback to the underlying promise's life cycles and, hance also returning the promise in this case.

[refack]

With latest comments incorporated IMHO two options have a meaningful use case

1. project - when the expected consumer is a user of a mixed API and expects uniform callback signatures (where the wrapped awaitable is a part of an API made by 3rd party)
2. catch - opt-in to incorporate the callback into the lifecycle of the underlying awaitable, i.e. having

   await util.callbackify(RunAsync, {catch: ture})((err, data) => {
      console.log("1");
   });
   console.log("2");

   print 1, 2

[benjamingr]

@refack catch should be on my default (otherwise what, no errors?) and project can just be something you call on the function later.

I think both of these still complicate the PR beyond scope, and may be added later if we really want them for some reason.

[addaleax]

@refack Can you backport (together with 873e2f2)?