Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add guide: "Backpressuring in Streams" #1109

Merged
merged 6 commits into from
Apr 22, 2017
Merged

Add guide: "Backpressuring in Streams" #1109

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e646720
Add guide: "Backpressuring in Streams"
jalafel Jan 23, 2017
26bbb66
review changes and update empty fields
jalafel Feb 26, 2017
3119d13
additional changes from review
jalafel Mar 4, 2017
c62f252
grammar fixes
jalafel Apr 7, 2017
56bdcdb
add ._writev pattern
jalafel Apr 7, 2017
d78008e
highwatermark default
jalafel Apr 14, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 45 additions & 6 deletions locale/en/docs/guides/backpressuring-in-streams.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,6 +298,12 @@ Well the answer is simple: Node.js does all of this automatically for you.
That's so great! But also not so great when we are trying to understand how to
implement our own custom streams.

Note: In most machines, there is a byte size that is determines when a buffer
is full (which will vary across different machines). Node.js allows you to set
your own custom [`highWaterMark`][], but commonly, the default is the optimal
value for what system is running the application. In instances where you might
want to raise that value, go for it, but do so with caution!
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you should write here the defaults, it usually helps people


## Lifecycle of `.pipe()`

To achieve a better understanding of backpressure, here is a flow-chart on the
Expand Down Expand Up @@ -476,13 +482,42 @@ class MyWritable extends Writable {
callback();
```

There are also some things to look out for when implementing [`._writev()`][].
The function is coupled with [`.cork()`][], but there is a common mistake when
writing:
```javascript
// Using .uncork() twice here makes two calls on the C++ layer, rendering the
// cork/uncork technique useless.
ws.cork()
ws.write('hello ')
ws.write('world ')
ws.uncork()

ws.cork()
ws.write('from ')
ws.write('Matteo')
ws.uncork()

// The correct way to write this is to utilize process.nextTick(), which fires
// on the next event loop.
ws.cork()
ws.write('hello ')
ws.write('world ')
process.nextTick(doUncork, ws)

ws.cork()
ws.write('from ')
ws.write('Matteo')
process.nextTick(doUncork, ws)

// as a global function
function doUncork (stream) {
stream.uncork()
}
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need an equal section on 'Write a Readable Stream'.

We should mention that [readable-stream][https://github.com/nodejs/readable-stream] is the best way to build a stream application across multiple versions of Node.js, and the browser too! This will make your application have a stable base, and an implementation that changes with its own lifecycle compared to node core.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a general question, is the primarily Node.js-based Readable stream still the best option in the browser? In my experience, the most often cause of using Node.js Readable in the browser is as a shim when creating Browserify bundle. At the same time, readable-stream IIRC depends on additional polyfills like one for events module. Plus, I know WHATWG has been working on their version of streams.

Regarding using the userland module in Node.js, definitely. You may also want to link to @rvagg's blog post

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

readable-stream is the way to go in the browser. readable-stream exports Readable & co, so all that is written here is valid there as well.

By using streams in the browser, I mean mostly to leverage code already written elsewhere for the purpose, i.e. the streams ecosystem.


Note: In most machines, there is a byte size that is determines when a buffer
is full (which will vary across different machines). Node.js allows you to set
your own custom [`highWaterMark`][], but commonly, the default is the optimal
value for what system is running the application. In instances where you might
want to raise that value, go for it, but do so with caution!

[`.cork()`][] can be called as many times we want, we just need to be careful to
call [`.uncork()`][] the same amount of times to make it flow again.

## Conclusion

Expand Down Expand Up @@ -513,6 +548,10 @@ Node.js.
[`.write()`]: https://nodejs.org/api/stream.html#stream_writable_write_chunk_encoding_callback
[`._read()`]: https://nodejs.org/docs/latest/api/stream.html#stream_readable_read_size_1
[`._write()`]: https://nodejs.org/docs/latest/api/stream.html#stream_writable_write_chunk_encoding_callback_1
[`._writev()`]: https://nodejs.org/api/stream.html#stream_writable_writev_chunks_callback
[`.cork()`]: https://nodejs.org/api/stream.html#stream_writable_cork
[`.uncork()`]: https://nodejs.org/api/stream.html#stream_writable_uncork

[push method]: https://nodejs.org/docs/latest/api/stream.html#stream_readable_push_chunk_encoding

[implementing Writable streams]: https://nodejs.org/docs/latest/api/stream.html#stream_implementing_a_writable_stream
Expand Down