Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StringDecoder undocumented size limit #35676

Closed
TanninOne opened this issue Oct 16, 2020 · 6 comments
Closed

StringDecoder undocumented size limit #35676

TanninOne opened this issue Oct 16, 2020 · 6 comments
Labels
confirmed-bug Issues with confirmed bugs. string_decoder Issues and PRs related to the string_decoder subsystem.

Comments

@TanninOne
Copy link

  • Version: v12.13.0
  • Platform: Windows 10 (64bit)
  • Subsystem: string_decoder, stream

What steps will reproduce the bug?

const buf = Buffer.alloc(1024 * 1024 * 1024, 'a');
const sd = new (require('string_decoder').StringDecoder)();
sd.write(buf).length

How often does it reproduce? Is there a required condition?

The above causes the issue 100% reliably for me.

What is the expected behavior?

Either the decoded buffer or an exception explaining that the input buffer is too long.

What do you see instead?

TypeError: Cannot read property 'length' of undefined

So sd.write(large buffer) returns undefined

The reason I'm including stream as an affected subsystem is this line which originally led me to this issue: https://github.com/nodejs/readable-stream/blob/040b813e60ecf0d68ac1461a3fc3157ea5785950/lib/_stream_readable.js#L277

So what happens is that if you have a stream (a pipe used for IPC in my case) and the other side sends an large chunk of data, the recipient throws an exception before any application code is invoked that could handle it gracefully.

Additional information

I don't know exactly how large the buffer has to be. In my application (electron 8.5.2 with node 12.13.0) this started happening at around 600MB, when reproducing in node on the command line I had to increase the buffer to 1GB.

Just to clarify: I understand that there will be a limit on how large these buffers can get, I just wish it would report an error that my application code can handle because right now I don't see how I could write robust client code that can deal with the other side sending crap.

@watilde watilde added the string_decoder Issues and PRs related to the string_decoder subsystem. label Oct 16, 2020
@targos
Copy link
Member

targos commented Dec 28, 2020

The size limit is exactly 0x1fffffe8 (536870888) characters. That's the maximum length a JS string can have in the V8 engine. The method correctly throws an error if the decoder is setup with ascii encoding. I'm looking into why that doesn't happen with utf-8

@targos
Copy link
Member

targos commented Dec 28, 2020

Well, I don't know why there's no exception propagating to JS...

Relevant lines in the code:

node/src/string_decoder.cc

Lines 265 to 268 in d146b25

MaybeLocal<String> ret =
decoder->DecodeData(args.GetIsolate(), content.data(), &length);
if (!ret.IsEmpty())
args.GetReturnValue().Set(ret.ToLocalChecked());

node/src/string_decoder.cc

Lines 208 to 211 in d146b25

if (nread > 0) {
if (!MakeString(isolate, data, nread, Encoding()).ToLocal(&body))
return MaybeLocal<String>();
} else {

if (encoding == UTF8) {
return String::NewFromUtf8(
isolate,
data,
v8::NewStringType::kNormal,
length);
} else {

/cc @addaleax

@targos
Copy link
Member

targos commented Dec 28, 2020

It's like String::NewFromUtf8 returns an empty MaybeLocal<String> but without a pending exception.

@addaleax
Copy link
Member

It's like String::NewFromUtf8 returns an empty MaybeLocal<String> but without a pending exception.

Yeah, it does – that’s a long-standing TODO in the V8 source… we should probably throw one ourselves :/

@targos
Copy link
Member

targos commented Dec 28, 2020

I see... Working on it!

@targos targos added the confirmed-bug Issues with confirmed bugs. label Dec 28, 2020
targos added a commit to targos/node that referenced this issue Dec 28, 2020
String::NewFromUtf8 doesn't generate an exception in V8 when the string
is too long but is guaranteed to return an empty MaybeLocal only in
that case. Generate a Node.js exception when it happens.

Fixes: nodejs#35676
@targos
Copy link
Member

targos commented Dec 28, 2020

PR at #36661

danielleadams pushed a commit that referenced this issue Jan 12, 2021
String::NewFromUtf8 doesn't generate an exception in V8 when the string
is too long but is guaranteed to return an empty MaybeLocal only in
that case. Generate a Node.js exception when it happens.

Fixes: #35676

PR-URL: #36661
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Gireesh Punathil <[email protected]>
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: James M Snell <[email protected]>
targos added a commit that referenced this issue May 25, 2021
String::NewFromUtf8 doesn't generate an exception in V8 when the string
is too long but is guaranteed to return an empty MaybeLocal only in
that case. Generate a Node.js exception when it happens.

Fixes: #35676

PR-URL: #36661
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Gireesh Punathil <[email protected]>
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: James M Snell <[email protected]>
targos added a commit that referenced this issue Jun 5, 2021
String::NewFromUtf8 doesn't generate an exception in V8 when the string
is too long but is guaranteed to return an empty MaybeLocal only in
that case. Generate a Node.js exception when it happens.

Fixes: #35676

PR-URL: #36661
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Gireesh Punathil <[email protected]>
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: James M Snell <[email protected]>
targos added a commit that referenced this issue Jun 11, 2021
String::NewFromUtf8 doesn't generate an exception in V8 when the string
is too long but is guaranteed to return an empty MaybeLocal only in
that case. Generate a Node.js exception when it happens.

Fixes: #35676

PR-URL: #36661
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Gireesh Punathil <[email protected]>
Reviewed-By: Rich Trott <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
confirmed-bug Issues with confirmed bugs. string_decoder Issues and PRs related to the string_decoder subsystem.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants