stream: convert string to Buffer when calling `unshift(<string>)` #27194

marcosc90 · 2019-04-11T22:29:24Z

readable.unshift can take a string as an argument, but that
string wasn't being converted to a Buffer, which caused a
TypeError: Argument must be a buffer in some cases.
A second optional argument encoding was added to unshift to
prevent all strings being coerced to utf8.

Fixes: #27192

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

A documentation change will be needed, for the encoding argument, but wanted to submit the PR first to see if it will get approved. I will also make a few tests for this if everything is okay.

Trott · 2019-04-11T23:04:12Z

Can you add a test for this?

marcosc90 · 2019-04-12T03:28:06Z

@Trott added multiple cases, if something is missing or needs to be changed let me know.

Trott · 2019-04-12T16:31:42Z

Thanks! Only other thing to add that I can see is the encoding argument to the documentation in doc/api/stream.md.

R: @nodejs/streams

mcollina · 2019-04-12T23:16:17Z

I think this is semver-minor as it's adding a new parameter.

CI: https://ci.nodejs.org/job/node-test-pull-request/22398/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1821/

nodejs-github-bot · 2019-04-12T23:16:32Z

CI: https://ci.nodejs.org/job/node-test-pull-request/22398/

addaleax

Can you also add a test for how this interacts with .setEncoding()?

marcosc90 · 2019-04-12T23:31:54Z

Can you also add a test for how this interacts with .setEncoding()?

I'll submit one in a few. Forgot about that scenario, thanks!

marcosc90 · 2019-04-13T00:12:38Z

@addaleax When doing that test, I remember that when state.decoder is set, the chunks are not converted to Buffer. The bug reported in #27192 won't happen, because when state.decoder is set, buffer.concat won't be used.

I don't think this PR should change that, I'm not even sure if it should be saved as a Buffer in that case, even though it's called BufferList, but I'm not very familiar with string decoder.

So I can test that the behaviour is unchanged, meaning that they're saved as string, and that data is emitted with the encoding set in .setEncoding.

Is that ok?

marcosc90 · 2019-04-13T02:02:50Z

There is one change left, but would like some feedback.

https://github.com/nodejs/node/blob/13492ba631a0a4aac0e34adcf0f6cfa3cf4e19bd/lib/_stream_readable.js#L229-L237

When .setEncoding is set and a chunk is .pushed, the convertion to that encoding is handled by the state.decoder, of course I can't use that same state.decoder for .unshifting otherwise it will mess with the internal buffer.

But in order to keep the same behaviour, I have to convert the pushed string, to the encoding defined by setEncoding, so the solution that I have right now working is the following:

if (addToFront && state.encoding && state.encoding !== encoding) {
  chunk = Buffer.from(chunk, encoding).toString(state.encoding);
} else if (encoding !== state.encoding) {
  chunk = Buffer.from(chunk, encoding);
  encoding = '';
}

Because the string must be saved in BufferList in the same encoding as setEncoding, so then is emitted to data that way.

Thoughts? Maybe someone has another alternative.

test/parallel/test-stream-readable-unshift.js

marcosc90 · 2019-04-24T00:40:06Z

Is there something else that should be added/changed?

mcollina

I’ve left some notes related to the tests. I think we should remove the usage of private API as much as possible - the majority of this feature should be testable without it.

mcollina · 2019-04-24T05:55:22Z

test/parallel/test-stream-readable-unshift.js

+    assert.strictEqual(
+      readable._readableState.buffer.head.data,
+      chunk
+    );


This test can be rewritten without using internals, it should consume the stream and check that the chunks comes out in order.

test/parallel/test-stream-readable-unshift.js

mcollina · 2019-04-24T05:58:22Z

test/parallel/test-stream-readable-unshift.js

+  assert.strictEqual(
+    readable._readableState.buffer.head.data.toString('utf8'),
+    string
+  );


This can be tested by consuming the stream and verifying that the data comes out in order.

mcollina

LGTM

nodejs-github-bot · 2019-06-02T14:22:29Z

CI: https://ci.nodejs.org/job/node-test-pull-request/23564/

`readable.unshift` can take a string as an argument, but that string wasn't being converted to a Buffer, which caused a <TypeError: Argument must be a buffer> in some cases. Also if a string was passed, that string was coerced to utf8 encoding. A second optional argument `encoding` was added to `unshift` to fix the encoding issue. Fixes: #27192 PR-URL: #27194 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Rich Trott <[email protected]>

Trott · 2019-06-02T14:57:27Z

Landed in df339bc

`readable.unshift` can take a string as an argument, but that string wasn't being converted to a Buffer, which caused a <TypeError: Argument must be a buffer> in some cases. Also if a string was passed, that string was coerced to utf8 encoding. A second optional argument `encoding` was added to `unshift` to fix the encoding issue. Fixes: #27192 PR-URL: #27194 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Rich Trott <[email protected]>

Notable changes: * doc: * The JSON variant of the API documentation is no longer experimental (Rich Trott) #27842. * esm: * JSON module support is always enabled under `--experimental-modules`. The `--experimental-json-modules` flag has been removed (Myles Borins) #27752. * http,http2: * A new flag has been added for overriding the default HTTP server socket timeout (which is two minutes). Pass `--http-server-default-timeout=milliseconds` or `--http-server-default-timeout=0` to respectively change or disable the timeout. Starting with Node.js 13.0.0, the timeout will be disabled by default (Ali Ijaz Sheikh) #27704. * inspector: * Added an experimental `--heap-prof` flag to start the V8 heap profiler on startup and write the heap profile to disk before exit (Joyee Cheung) #27596. * stream: * The `readable.unshift()` method now correctly converts strings to buffers. Additionally, a new optional argument is accepted to specify the string's encoding, such as `'utf8'` or `'ascii'` (Marcos Casagrande) #27194. * v8: * The object returned by `v8.getHeapStatistics()` has two new properties: `number_of_native_contexts` and `number_of_detached_contexts` (Yuriy Vasiyarov) #27933. PR-URL: #28040

Notable changes: * doc: * The JSON variant of the API documentation is no longer experimental (Rich Trott) nodejs#27842. * esm: * JSON module support is always enabled under `--experimental-modules`. The `--experimental-json-modules` flag has been removed (Myles Borins) nodejs#27752. * http,http2: * A new flag has been added for overriding the default HTTP server socket timeout (which is two minutes). Pass `--http-server-default-timeout=milliseconds` or `--http-server-default-timeout=0` to respectively change or disable the timeout. Starting with Node.js 13.0.0, the timeout will be disabled by default (Ali Ijaz Sheikh) nodejs#27704. * inspector: * Added an experimental `--heap-prof` flag to start the V8 heap profiler on startup and write the heap profile to disk before exit (Joyee Cheung) nodejs#27596. * stream: * The `readable.unshift()` method now correctly converts strings to buffers. Additionally, a new optional argument is accepted to specify the string's encoding, such as `'utf8'` or `'ascii'` (Marcos Casagrande) nodejs#27194. * v8: * The object returned by `v8.getHeapStatistics()` has two new properties: `number_of_native_contexts` and `number_of_detached_contexts` (Yuriy Vasiyarov) nodejs#27933. PR-URL: nodejs#28040

BethGriggs · 2019-09-03T15:27:23Z

Should this land on v10.x? //cc @mcollina

mcollina · 2019-09-04T11:18:57Z

Yes please!

nodejs-github-bot added the stream Issues and PRs related to the stream subsystem. label Apr 11, 2019

mcollina added the semver-minor PRs that contain new features and should be released in the next minor version. label Apr 12, 2019

addaleax approved these changes Apr 12, 2019

View reviewed changes

BridgeAR reviewed Apr 16, 2019

View reviewed changes

test/parallel/test-stream-readable-unshift.js Outdated Show resolved Hide resolved

mcollina reviewed Apr 24, 2019

View reviewed changes

mcollina approved these changes Apr 26, 2019

View reviewed changes

jasnell approved these changes Apr 26, 2019

View reviewed changes

This comment has been minimized.

Sign in to view

Trott added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jun 2, 2019

Trott approved these changes Jun 2, 2019

View reviewed changes

Trott merged commit df339bc into nodejs:master Jun 2, 2019

targos mentioned this pull request Jun 3, 2019

v12.4.0 release proposal #28040

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stream: convert string to Buffer when calling `unshift(<string>)` #27194

stream: convert string to Buffer when calling `unshift(<string>)` #27194

marcosc90 commented Apr 11, 2019 •

edited

Loading

Trott commented Apr 11, 2019

marcosc90 commented Apr 12, 2019

Trott commented Apr 12, 2019

mcollina commented Apr 12, 2019

nodejs-github-bot commented Apr 12, 2019

addaleax left a comment

marcosc90 commented Apr 12, 2019

marcosc90 commented Apr 13, 2019 •

edited

Loading

marcosc90 commented Apr 13, 2019 •

edited

Loading

marcosc90 commented Apr 24, 2019

mcollina left a comment

mcollina Apr 24, 2019

mcollina Apr 24, 2019

mcollina left a comment

This comment has been minimized.

nodejs-github-bot commented Jun 2, 2019

Trott commented Jun 2, 2019

BethGriggs commented Sep 3, 2019

mcollina commented Sep 4, 2019

stream: convert string to Buffer when calling unshift(<string>) #27194

stream: convert string to Buffer when calling unshift(<string>) #27194

Conversation

marcosc90 commented Apr 11, 2019 • edited Loading

Checklist

Trott commented Apr 11, 2019

marcosc90 commented Apr 12, 2019

Trott commented Apr 12, 2019

mcollina commented Apr 12, 2019

nodejs-github-bot commented Apr 12, 2019

addaleax left a comment

Choose a reason for hiding this comment

marcosc90 commented Apr 12, 2019

marcosc90 commented Apr 13, 2019 • edited Loading

marcosc90 commented Apr 13, 2019 • edited Loading

marcosc90 commented Apr 24, 2019

mcollina left a comment

Choose a reason for hiding this comment

mcollina Apr 24, 2019

Choose a reason for hiding this comment

mcollina Apr 24, 2019

Choose a reason for hiding this comment

mcollina left a comment

Choose a reason for hiding this comment

This comment has been minimized.

nodejs-github-bot commented Jun 2, 2019

Trott commented Jun 2, 2019

BethGriggs commented Sep 3, 2019

mcollina commented Sep 4, 2019

stream: convert string to Buffer when calling `unshift(<string>)` #27194

stream: convert string to Buffer when calling `unshift(<string>)` #27194

marcosc90 commented Apr 11, 2019 •

edited

Loading

marcosc90 commented Apr 13, 2019 •

edited

Loading

marcosc90 commented Apr 13, 2019 •

edited

Loading