buffer: convert offset & length to int properly #9492

thefourtheye · 2016-11-06T13:48:36Z

Checklist

make -j8 test (UNIX), or vcbuild test nosign (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

buffer

Description of change

As per ecma-262 2015's #sec-%typedarray%-buffer-byteoffset-length,
offset would be an integer, not a 32 bit unsigned integer. Also,
length would be an integer with the maximum value of 2^53 - 1, not a
32 bit unsigned integer.

This would be a problem because, if we create a buffer from an
arraybuffer, from an offset which is greater than 2^32, it would be
actually pointing to a different location in arraybuffer. For example,
if we use 2^40 as offset, then the actual value used will be 0,
because byteOffset >>>= 0 will convert byteOffset to a 32 bit
unsigned int, which is based on 2^32 modulo.

cc @nodejs/buffer

As per ecma-262 2015's #sec-%typedarray%-buffer-byteoffset-length, `offset` would be an integer, not a 32 bit unsigned integer. Also, `length` would be an integer with the maximum value of 2^53 - 1, not a 32 bit unsigned integer. This would be a problem because, if we create a buffer from an arraybuffer, from an offset which is greater than 2^32, it would be actually pointing to a different location in arraybuffer. For example, if we use 2^40 as offset, then the actual value used will be 0, because `byteOffset >>>= 0` will convert `byteOffset` to a 32 bit unsigned int, which is based on 2^32 modulo.

cjihrig · 2016-11-08T13:29:49Z

test/parallel/test-internal-util-toInteger.js

@@ -0,0 +1,19 @@
+// Flags: --expose-internals


This should probably test for a wider range of values. test/parallel/test-net-internal.js has some good examples.

thefourtheye · 2016-11-12T11:36:14Z

Ping!

cjihrig

LGTM, but someone from @nodejs/buffer should sign off.

thefourtheye · 2016-11-17T11:52:54Z

Bump!

jasnell

Small nit in the tests but otherwise LGTM

jasnell · 2016-11-18T16:43:30Z

test/parallel/test-buffer-creation-regression.js

+    uint8Array[i] = 1;
+  }
+
+  const buffer = new Buffer(arrayBuffer, offset, length);


s/new Buffer/Buffer.from

@jasnell Done!

trevnorris · 2016-11-23T13:12:56Z

lib/internal/util.js

+
+/*
+ * Implementation of ToLength as per ECMAScript Specification
+ * Refer: http://www.ecma-international.org/ecma-262/6.0/#sec-tolength


I'm curious why the step If len is +∞, return 253-1. is in there when the min() operation makes it seem unnecessary. I can't see any difference if that condition is left out.

True. In our implementation I omitted that, as min itself is sufficient.

@trevnorris I raised a PR tc39/ecma262#738. Let's see what tc39 people feel.

@trevnorris And it got merged :-)

@thefourtheye excellent!

thefourtheye · 2016-11-26T05:53:24Z

Landed in ca37fa5

As per ecma-262 2015's #sec-%typedarray%-buffer-byteoffset-length, `offset` would be an integer, not a 32 bit unsigned integer. Also, `length` would be an integer with the maximum value of 2^53 - 1, not a 32 bit unsigned integer. This would be a problem because, if we create a buffer from an arraybuffer, from an offset which is greater than 2^32, it would be actually pointing to a different location in arraybuffer. For example, if we use 2^40 as offset, then the actual value used will be 0, because `byteOffset >>>= 0` will convert `byteOffset` to a 32 bit unsigned int, which is based on 2^32 modulo. PR-URL: #9492 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Trevor Norris <[email protected]>

Trott · 2016-11-26T17:50:46Z

It looks like this landed without being run through CI. This is a guess right now, so I could be wrong, but it appears this is the likely culprit in having broken CI. See https://ci.nodejs.org/job/node-test-commit-arm/6094/nodes=armv7-ubuntu1404/console and https://ci.nodejs.org/job/node-test-commit-linux/6250/nodes=centos5-32/console and a dozen or two others:

not ok 24 parallel/test-buffer-creation-regression
  ---
  duration_ms: 0.248
  severity: fail
  stack: |-
    /home/iojs/build/workspace/node-test-commit-linux/nodes/centos5-32/test/parallel/test-buffer-creation-regression.js:7
      const arrayBuffer = new ArrayBuffer(size);
                          ^
    
    RangeError: Invalid array buffer length
        at new ArrayBuffer (<anonymous>)
        at test (/home/iojs/build/workspace/node-test-commit-linux/nodes/centos5-32/test/parallel/test-buffer-creation-regression.js:7:23)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux/nodes/centos5-32/test/parallel/test-buffer-creation-regression.js:21:1)
        at Module._compile (module.js:571:32)
        at Object.Module._extensions..js (module.js:580:10)
        at Module.load (module.js:488:32)
        at tryModuleLoad (module.js:447:12)
        at Function.Module._load (module.js:439:3)
        at Module.runMain (module.js:605:10)
        at run (bootstrap_node.js:420:7)

Trott · 2016-11-26T17:52:14Z

Update: Oh, yeah, that test was added in this PR so yeah, all but certainly this change is the problem. Is there a quick fix? Or should we revert and take whatever time is needed to fix it?

Trott · 2016-11-26T17:56:18Z

Opened #9814 for a quick revert. Can always re-do the change after it's fixed. PTAL over there...

This reverts commit ca37fa5. A test provided by the commit fails on most (but not all) platforms on CI. PR-URL: nodejs#9814 Ref: nodejs#9492 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Сковорода Никита Андреевич <[email protected]>

Trott · 2016-11-26T19:43:22Z

Revert for this landed in 6b2aa1a. Hopefully the fix is easy enough and this can go back in soon. /cc @thefourtheye

brodycj · 2016-11-26T20:23:26Z

Also test\parallel\test-buffer-creation-regression.js with this change gave me a RangeError exception on Windows.

Path: parallel/test-buffer-creation-regression                                                                          
C:\Users\Chris\Documents\work\node\test\parallel\test-buffer-creation-regression.js:7
  const arrayBuffer = new ArrayBuffer(size);
                      ^

RangeError: Invalid array buffer length
    at new ArrayBuffer (<anonymous>)
    at test (C:\Users\Chris\Documents\work\node\test\parallel\test-buffer-creation-regression.js:7:23)
    at Object.<anonymous> (C:\Users\Chris\Documents\work\node\test\parallel\test-buffer-creation-regression.js:21:1)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.runMain (module.js:605:10)
    at run (bootstrap_node.js:420:7)
Command: C:\Users\Chris\Documents\work\node\Release\node.exe C:\Users\Chris\Documents\work\node\test\parallel\test-buffer-creation-regression.js

As per ecma-262 2015's #sec-%typedarray%-buffer-byteoffset-length, `offset` would be an integer, not a 32 bit unsigned integer. Also, `length` would be an integer with the maximum value of 2^53 - 1, not a 32 bit unsigned integer. This would be a problem because, if we create a buffer from an arraybuffer, from an offset which is greater than 2^32, it would be actually pointing to a different location in arraybuffer. For example, if we use 2^40 as offset, then the actual value used will be 0, because `byteOffset >>>= 0` will convert `byteOffset` to a 32 bit unsigned int, which is based on 2^32 modulo. This is a redo, as the ca37fa5 broke CI. Refer: nodejs#9814 Refer: nodejs#9492

As per ecma-262 2015's #sec-%typedarray%-buffer-byteoffset-length, `offset` would be an integer, not a 32 bit unsigned integer. Also, `length` would be an integer with the maximum value of 2^53 - 1, not a 32 bit unsigned integer. This would be a problem because, if we create a buffer from an arraybuffer, from an offset which is greater than 2^32, it would be actually pointing to a different location in arraybuffer. For example, if we use 2^40 as offset, then the actual value used will be 0, because `byteOffset >>>= 0` will convert `byteOffset` to a 32 bit unsigned int, which is based on 2^32 modulo. This is a redo, as the ca37fa5 broke CI. Refer: #9814 Refer: #9492 PR-URL: #9815 Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Trevor Norris <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Matteo Collina <[email protected]>

As per ecma-262 2015's #sec-%typedarray%-buffer-byteoffset-length, `offset` would be an integer, not a 32 bit unsigned integer. Also, `length` would be an integer with the maximum value of 2^53 - 1, not a 32 bit unsigned integer. This would be a problem because, if we create a buffer from an arraybuffer, from an offset which is greater than 2^32, it would be actually pointing to a different location in arraybuffer. For example, if we use 2^40 as offset, then the actual value used will be 0, because `byteOffset >>>= 0` will convert `byteOffset` to a 32 bit unsigned int, which is based on 2^32 modulo. This is a redo, as the ca37fa5 broke CI. Refer: nodejs#9814 Refer: nodejs#9492 PR-URL: nodejs#9815 Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Trevor Norris <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Matteo Collina <[email protected]>

As per ecma-262 2015's #sec-%typedarray%-buffer-byteoffset-length, `offset` would be an integer, not a 32 bit unsigned integer. Also, `length` would be an integer with the maximum value of 2^53 - 1, not a 32 bit unsigned integer. This would be a problem because, if we create a buffer from an arraybuffer, from an offset which is greater than 2^32, it would be actually pointing to a different location in arraybuffer. For example, if we use 2^40 as offset, then the actual value used will be 0, because `byteOffset >>>= 0` will convert `byteOffset` to a 32 bit unsigned int, which is based on 2^32 modulo. PR-URL: nodejs#9492 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Trevor Norris <[email protected]>

This reverts commit ca37fa5. A test provided by the commit fails on most (but not all) platforms on CI. PR-URL: nodejs#9814 Ref: nodejs#9492 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Сковорода Никита Андреевич <[email protected]>

As per ecma-262 2015's #sec-%typedarray%-buffer-byteoffset-length, `offset` would be an integer, not a 32 bit unsigned integer. Also, `length` would be an integer with the maximum value of 2^53 - 1, not a 32 bit unsigned integer. This would be a problem because, if we create a buffer from an arraybuffer, from an offset which is greater than 2^32, it would be actually pointing to a different location in arraybuffer. For example, if we use 2^40 as offset, then the actual value used will be 0, because `byteOffset >>>= 0` will convert `byteOffset` to a 32 bit unsigned int, which is based on 2^32 modulo. This is a redo, as the ca37fa5 broke CI. Refer: #9814 Refer: #9492 PR-URL: #9815 Reviewed-By: Michaël Zasso <[email protected]> Reviewed-By: Trevor Norris <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Backport-Of: #9815 PR-URL: #11176 Reviewed-By: James M Snell <[email protected]>

thefourtheye added the buffer Issues and PRs related to the buffer subsystem. label Nov 6, 2016

nodejs-github-bot added buffer Issues and PRs related to the buffer subsystem. util Issues and PRs related to the built-in util module. labels Nov 6, 2016

cjihrig reviewed Nov 8, 2016

View reviewed changes

simplify implementation and improve tests

241d32a

cjihrig approved these changes Nov 12, 2016

View reviewed changes

jasnell approved these changes Nov 18, 2016

View reviewed changes

change to Buffer.from

c5f868a

trevnorris reviewed Nov 23, 2016

View reviewed changes

trevnorris approved these changes Nov 25, 2016

View reviewed changes

thefourtheye closed this Nov 26, 2016

thefourtheye deleted the buffer-from-array-buffer branch November 26, 2016 05:53

Trott mentioned this pull request Nov 26, 2016

Revert "buffer: convert offset & length to int properly" #9814

Closed

1 task

thefourtheye mentioned this pull request Nov 27, 2016

buffer: convert offset & length to int properly #9815

Closed

3 tasks

addaleax added dont-land-on-v4.x labels Dec 5, 2016

thefourtheye mentioned this pull request Feb 5, 2017

Backport 9815 & 10161 to v6.x #11176

Closed

3 tasks

jasnell mentioned this pull request Apr 4, 2017

8.0.0 Release Proposal #12220

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buffer: convert offset & length to int properly #9492

buffer: convert offset & length to int properly #9492

thefourtheye commented Nov 6, 2016

cjihrig Nov 8, 2016

thefourtheye Nov 12, 2016

thefourtheye commented Nov 12, 2016

cjihrig left a comment

thefourtheye commented Nov 17, 2016

jasnell left a comment

jasnell Nov 18, 2016

thefourtheye Nov 20, 2016

trevnorris Nov 23, 2016

thefourtheye Nov 23, 2016

thefourtheye Nov 23, 2016

thefourtheye Nov 23, 2016

trevnorris Nov 25, 2016

thefourtheye commented Nov 26, 2016

Trott commented Nov 26, 2016

Trott commented Nov 26, 2016

Trott commented Nov 26, 2016

Trott commented Nov 26, 2016

brodycj commented Nov 26, 2016

buffer: convert offset & length to int properly #9492

buffer: convert offset & length to int properly #9492

Conversation

thefourtheye commented Nov 6, 2016

Checklist

Affected core subsystem(s)

Description of change

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thefourtheye commented Nov 12, 2016

cjihrig left a comment

Choose a reason for hiding this comment

thefourtheye commented Nov 17, 2016

jasnell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thefourtheye commented Nov 26, 2016

Trott commented Nov 26, 2016

Trott commented Nov 26, 2016

Trott commented Nov 26, 2016

Trott commented Nov 26, 2016

brodycj commented Nov 26, 2016