-
Notifications
You must be signed in to change notification settings - Fork 609
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[node-core-library] Provide async utilities in core library #2665
Conversation
/** | ||
* Return a promise that resolves after the specified number of milliseconds. | ||
*/ | ||
public static async sleep(ms: number): Promise<void> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be useful for this to accept a cancellation token, but we can add that later.
public static async mapLimitAsync<TEntry, TRetVal>( | ||
array: TEntry[], | ||
parallelismLimit: number, | ||
fn: ((entry: TEntry) => Promise<TRetVal>) | ((entry: TEntry, index: number) => Promise<TRetVal>) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fn: ((entry: TEntry) => Promise<TRetVal>) | ((entry: TEntry, index: number) => Promise<TRetVal>) | |
fn: (entry: TEntry, index: number) => Promise<TRetVal> |
Someone can choose to just ignore the index parameter.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't this error if you pass a callback to the function that doesn't have an index param?
I guess to me this seems pretty unfriendly and veers further from the typical Array.map / Array.forEach behaviors, which I like to emulate because they are the baseline people are familiar with.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If that's what the Array.map
typings say, I guess just do that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I reacted too early to an error I was seeing and made an assumption -- your example looks good, and simpler is probably better! I'll use the one without the algebra.
public static async forEachLimitAsync<TEntry>( | ||
array: TEntry[], | ||
parallelismLimit: number, | ||
fn: ((entry: TEntry) => Promise<void>) | ((entry: TEntry, index: number) => Promise<void>) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fn: ((entry: TEntry) => Promise<void>) | ((entry: TEntry, index: number) => Promise<void>) | |
fn: (entry: TEntry, index: number) => Promise<void> |
parallelismLimit: number, | ||
fn: ((entry: TEntry) => Promise<void>) | ((entry: TEntry, index: number) => Promise<void>) | ||
): Promise<void> { | ||
return new Promise((resolve: () => void, reject: (error: Error) => void) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return new Promise((resolve: () => void, reject: (error: Error) => void) => { | |
await new Promise((resolve: () => void, reject: (error: Error) => void) => { |
* Return a promise that resolves after the specified number of milliseconds. | ||
*/ | ||
public static async sleep(ms: number): Promise<void> { | ||
return new Promise((resolve) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return new Promise((resolve) => { | |
await new Promise((resolve) => { |
while (operationsInProgress < parallelismLimit) { | ||
if (index < array.length) { | ||
operationsInProgress++; | ||
fn(array[index], index++) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happens if fn
throws a synchronous exception?
After a long conversation with @octogonz, he's convinced me that it might be better to tweak the API for the async utilities.
In addition to the benefits of familiarity, having an options object that can be extended later gives a nice place to stash features that might come up (perhaps we want a different type of rate-limiting, like a per-second or debounce type behavior, instead of a concurrency limit; or maybe we want a boolean to control whether an error short-circuits the rest of the list or is guaranteed to execute all operations, etc.). One other change I made is to add a little bit of protection against a synchronous function / synchronous throw. You shouldn't be able to encounter this if you're writing valid TypeScript, but you could imagine for example, someone writing a repo-task tool and using Last, I changed the definition from |
common/changes/@rushstack/node-core-library/node-core-async_2021-04-30-11-01.json
Outdated
Show resolved
Hide resolved
…21-04-30-11-01.json Co-authored-by: Pete Gonzalez <[email protected]>
Summary
I needed a "map limit" implementation and did not see one in the existing codebase; however, @iclanton recently added a "forEach limit" implementation to the
heft
source.This PR moves that function over to
node-core-library
instead so other projects can use it, and adds a couple other commonly used asynchronous functions.(I'm not the first one to have this idea... looks like it was discussed before in pending PR #2431, although in that case it was exporting from
heft
.)Details
forEachLimitAsync
over to a new module,Async
, fromheft
.mapLimitAsync
.index
for the callback function (to match the signature ofArray#map
andArray#forEach
).Async.sleep
. (In JavaScript I don't really bother, it's easy to make an inline setTimeout, but it's nice to have the method w/signature around for TypeScript in my opinion.)The previous implementation of
forEachLimitAsync
has been left inheft
and just calls through to the core library now (this is an area I could potentially clean up more).@octogonz suggested an alternate approach here, which is to bring in a new third party library that does this. I often pull bottleneck into applications I'm working on, for example, to provide generic rate limiting. I'm open to that approach as well, but given that the guts were already implemented, this PR seems the easier option.
I've marked the new
Async
export as@public
-- making this available to monorepo maintainers for use inrepo-tasks
, etc. (and not just for rushstack internal repo) seems pretty useful.How it was tested
forEachLimitAsync
andmapLimitAsync
, to validate basic behavior.