V8 Changes - HasIndexedPropertiesInExternalArrayData, GetIndexedPropertiesExternalArrayDataLength, GetIndexedPropertiesExternalArrayData no longer available - what to use instead?

[metabench]

Since v2.2.0 V8 has changed in a way so that my addon would no longer compile.

This would work previously:

Local<Object> obj = args[0].As<Object>();
if (obj->HasIndexedPropertiesInExternalArrayData()) {
    arr_data_structure_length = obj->GetIndexedPropertiesExternalArrayDataLength();
    arr = static_cast<double*>(obj->GetIndexedPropertiesExternalArrayData());
}

Now I get errors such as

 error C2039: 'HasIndexedPropertiesInExternalArrayData' : is not a member of 'v8::Object'
 error C2039: 'GetIndexedPropertiesExternalArrayDataLength' : is not a member of 'v8::Object'
 error C2039: 'GetIndexedPropertiesExternalArrayData' : is not a member of 'v8::Object'

How should it be done now?

[rvagg]

Probably one for @trevnorris to explain how it was dealt with for Buffer and what to look at for examples. Basically we've had to use V8 TypedArrays (Uint8Array) for this which may also be possible for your use.

[bnoordhuis]

Haven't you asked this question before in #883?

[metabench]

@bnoordhuis I asked about how to get access to data as arrays, which indeed is the same goal I have here. Since then, V8 has changed, so what I found out then no longer applies. Smalloc is out of the picture too now.

[bnoordhuis]

Okay. The basic premise is still the same: you use an ArrayBuffer for storage and you can optionally put a typed array like Uint8Array on top of that. External array data as a feature on arbitrary JS objects is completely gone now.

Have a look at deps/v8/include/v8.h, particularly the ArrayBuffer, ArrayBufferView and TypedArray classes. If you have specific questions, I'll try to answer them.

[trevnorris]

Key things to watch out for:

• When creating the ArrayBuffer, and you want to pre-allocate the memory, if you want v8 to control the lifetime of the memory will need to create with kInternalized.
• Use ArrayBuffer::GetContents() if you want to retrieve the data without externalizing the contents.

[mikeseven]

@bnoordhuis @trevnorris so this means there is no way to access to a pointer to JS Array data anymore? At least, I came to this conclusion. For node bindings to OpenCL (node-opencl), this implies only supporting ArrayBuffer and TypedArrays, which are more optimal anyway.
Just want to make sure I'm not missing something.
Thanks.

[trevnorris]

so this means there is no way to access to a pointer to JS Array data anymore?

Don't think I fully understand the question, but everything that could be done with the previous implementation can still be done today. I would simply recommend using the node_buffer.h API so you don't have to worry about it. i.e. node::Buffer::Data(Local<Value>) and node::Buffer::Length(Local<Value>).

[mikeseven]

Not really. Doing that would make a copy of the data from JS Array to node::Buffer. This could be OK if the buffer is for read only. But if it is for write, then you'll have to do another copy back to the JS Array.

[metabench]

The current use case I have in mind involves creating Typed Arrays within JavaScript, and then both reading and writing them within C++. Would someone who knows more about the underlying V8 implementation please advise me on if this still possible through casting the typed array to an array of C++ native types?

@trevnorris Buffers may be a good way to approach this if I were to be writing new code (though maybe Typed Arrays would in fact work better), but in this case I have existing code that's using Typed Arrays and would like easy and C++ idiomatic read/write access to them.

@bnoordhuis Thanks. I have not yet had a chance to study those V8 files. It's very useful to know what to refer to in the near future. I do have one question right now though. I'll clarify by saying that I'm only interested in getting read/write access to Typed Arrays. Is that still possible to do using pointers with predictable indexes, or has the underlying V8 engine moved away from that structure?

[targos]

@metabench You may find some interesting things in the code of node-canvas. I think this line answers your question about getting a pointer to the Typed Array data.

[trevnorris]

@mikeal by "JS Array" do you mean the indexed object returned by new Buffer()? That has me confused, because Array.isArray(new Buffer(n)) == false.

There's been an alteration in the native buffer API. Which is node::Buffer::New() takes control of the data when a char* is passed. The memory copy API has been moved to node::Buffer::Copy(). So accessing node::Buffer::Data() will return a char* to the memory located in the buffer object.

@metabench Buffer is now a Uint8Array. Here's part of our implementation that creates a new Uint8Array from existing memory and turns it into a Buffer: https://github.com/nodejs/node/blob/v4.1.1/src/node_buffer.cc#L375-L392

[mikeal]

@trevnorris I think you meant @mikeseven :)

[bnoordhuis]

I'm only interested in getting read/write access to Typed Arrays. Is that still possible to do using pointers with predictable indexes

I'm not really sure what you by 'pointers with predictable indexes'. If you mean if it's possible to create an ArrayBuffer from a pointer to a chunk of memory you own, yes, that's possible.

[metabench]

@bnoordhuis I was referring to the structure in which the data is arranged within the Typed Array. The items in the array are contiguous within memory in the older V8 that I am used to. By the looks of things this is also the case with the newer V8 implementation.

Thanks to everyone for the advice on how to solve this.

[martinheidegger]

I think the answer to this is:

type check

error C2039: 'HasIndexedPropertiesInExternalArrayData' : is not a member of 'v8::Object'

becomes

info[x]->Is..Array()
i.e.: info[0]->IsFloat32Array()

Note: This is experimental?! http://bespin.cz/~ondras/html/classv8_1_1Value.html#a4effc7ca1a221dd8c1e23c0f28145ef0

length

error C2039: 'GetIndexedPropertiesExternalArrayDataLength' : is not a member of 'v8::Object'

becomes

info[x].As<...Array>()->Length();
i.e.: info[0].As<Float32Array>()->Length()

data

error C2039: 'GetIndexedPropertiesExternalArrayData' : is not a member of 'v8::Object'

becomes

info[x]->As<...Array>()->Buffer()->GetContents().Data()
i.e.: info[0]->As<Float32Array>()->Buffer()->GetContents().Data()

[trevnorris]

@martinheidegger What are you using Float32Array for?

[martinheidegger]

@trevnorris andreasgal/node-kissfft#1

[trevnorris]

Okay. Just remember that the typed array doesn't hand you back a pointer to the data. You need to get contents from the array buffer the add the byte offset to get the pointer the typed array is using.

[martinheidegger]

@trevnorris Didn't understand the 2nd sentence at all....

[trevnorris]

Properly retrieving the char* from a typed array passed to C++:

  Local<Uint8Array> ui8 = args[0].As<Uint8Array>();
  ArrayBuffer::Contents ui8_c = ui8->Buffer()->GetContents();
  const size_t ui8_offset = ui8->ByteOffset();
  const size_t ui8_length = ui8->ByteLength();
  char* const ui8_data = static_cast<char*>(ui8_c.Data()) + ui8_offset;
  if (ui8_length > 0)
    assert(ui8_data != nullptr);

[Fishrock123]

Gona close, feel free to continue asking questions if necessary. :)

[metabench]

I have noticed that the latest version of Nan has got Nan::TypedArrayContents (https://github.com/nodejs/nan/blob/master/doc/v8_misc.md#nantypedarraycontents)

This looks like it is going to be the easiest way to access typed arrays in the long run, though I still could do with an example of how to use it. Perhaps I should ask in the Nan project though.

I want to do this with a low overhead and don't know quite how efficient Nan would be compared to coding it more directly. It does make sense that accessing a Typed Array is something that Nan will provide an abstraction to.

[blazs]

@martinheidegger and @trevnorris Why not just use node::Buffer::Data() for retrieving data and node::Buffer::Length() for retrieving length?

[trevnorris]

@blazs Because @martinheidegger identified a Float32Array was used. The Buffer methods only work on Uint8Array.

[blazs]

@trevnorris Ah, yes. Thanks.

[j-funk]

I think the answer to this is...

@martinheidegger thank you