JSI: Add no-copy ArrayBuffer constructor

[mrousavy]

Problem

I'm trying to efficiently use ArrayBuffers to avoid expensive copy operations. To expose buffers to JS I use this TypedArray implementation by Expo, which unfortunately copies the entire buffer (see code here). Because the entire buffer is copied, this causes two separate issues:

• Creating a new TypedArray (e.g. Uint8Array) is very slow because those huge buffers (~ 10 MB) have to be copied
• Doing this very often (in my case up to 240 times a second) causes out of memory errors since I cannot manually free the buffer when I know it is not needed anymore. (or at least re-use it)

---

Use-case 1

I'm the maintainer of VisionCamera which provides an API to run any sort of frame processing code for each frame the camera "sees". For example, you might use this to run some AI to detect faces, objects, barcodes or whatever.

Since the frame objects are so big, I tried to directly expose the native pixel buffer (~ 10 MB per frame) to JS by using ArrayBuffers. Those callbacks are called 30, 60 and sometimes even 240 times per second, so copying all of the pixels every time will not be fast enough - instead I want to directly use the buffer I already have in memory. Also I am experiencing out of memory errors since the JSI ArrayBuffer implementation maintains a custom buffer which is only cleaned when the JS VM runs a garbage collection - instead I either want to manually free the buffer, or re-use my custom buffer.

Working on this here: mrousavy/react-native-vision-camera#308

Use-case 2

I'm working on a feature to stream Audio Sample Buffers from a audio/music player in realtime where you can read those values in JS. Currently I use normal Arrays for this, but I want to improve the performance by avoiding copy operations and instead directly expose the native audio sample buffer array to JS using an ArrayBuffer. The problem here is as well that the ArrayBuffer implementation copies the entire buffer.

Working on this here: expo/expo#13516

---

So TL;DR: Currently the ArrayBuffer API does not provide a way to initialize a new ArrayBuffer from an already existing memory buffer without copying the entire thing.

Solution

Update the JSI API to provide an initializer for ArrayBuffer with the following signature:

ArrayBuffer(void* buffer, size_t size);

I am not aware of any workarounds that do not copy the entire buffer at the moment.

Additional Context

Here's the API from JSC to create an ArrayBuffer using an existing memory buffer: JSTypedArray.h: JSObjectMakeTypedArrayWithBytesNoCopy

afaik, Hermes does not provide this constructor yet.

There have been numerous feature requests and discussions for this:

• Add methods to create ArrayBuffers from byte arrays #419
• Add methods to create ArrayBuffers from byte arrays react-native#30445
• Add TypedArray support to jsi #182
• Add TypedArray support to jsi #248

[mrousavy]

cc @savv @tmikov @sercand @mhorowitz @wkozyra95 @ryantrem because you guys were part of the discussions in the previous issues/PRs - would love to get your opinion on this.

[ryantrem]

+@bghgary, and thanks for the ping @mrousavy.

It would be great to have this feature, and there are scenarios in Babylon React Native where we would use it, such as when we download files and expose them to JS (we have a JSI-based implementation of XmlHttpRequest which is much more efficient than the XmlHttpRequest included in React Native).

For the scenarios described above though, it's not 100% clear to me how much this will help. I would expect that for something like VisionCamera the following would be true:

• You are getting some kind of RGB buffer from OS APIs (either as a texture or some kind of raw memory buffer)
• You get those buffers on some thread that is related to camera capture (not the JS thread)
• Those buffers are re-used per frame (the OS does not allocate a 10mb buffer at 240 hz)
• Doing any kind of per-pixel operation on a 10mb RGB buffer at 240hz (or even 60hz) on the JS side is impractical from a performance standpoint (I don't think you could even iterate over the elements of a 10mb ArrayBuffer at 240hz on the JS side, let alone do any processing on the data - JS is slow without JIT)
• Doing processing on the 10mb ArrayBuffer probably needs to happen in native code then, possibly in another JSI-based native module

The above basically outlines the constraints we have in Babylon React Native for a similar scenario, where we have a frame capture feature that can get what is being rendered on screen by Babylon each frame and expose it as an RGB buffer to JS. We don't attempt to consume this ArrayBuffer in JS code though, we just pass it off to other JSI native modules (for things like encoding the captured frames into an mp4 in real time). In this case, there are three threads, and each one needs one copy of the buffer (plus one temp buffer, described below):

• The capture thread: in our case, this is our native renderer, which does not run on the JS thread; in your case, I'd think it would be the thread capturing and providing raw camera frames. Presumably there is one buffer/texture managed by the OS for this.
• The JS thread: in our case, we create a single ArrayBuffer once (re-used for each frame during capture), and each time we get a new frame on the capture thread, we copy the buffer to a temp buffer, dispatch to the JS thread, copy the temp buffer into the single reused ArrayBuffer, and give the JS a chance to do something with this buffer. The JS contract in this case is that the buffer has the data for the current frame during this JS invocation, but that data in the ArrayBuffer will change in future frames. I would expect you could do something similar.
• The frame processing thread: in our case, we don't want to block the JS thread while we process the captured frame, so this code is called from JS on the JS thread, but it immediately copies the ArrayBuffer into its own buffer and dispatches to a background thread to do the processing.

So we end up basically with three persistent buffers and one temp buffer, and three memcpys per frame, which is pretty manageable, and it can pretty easily be done at 60hz. The JS for the most part just passes around an ArrayBuffer between different JSI-based native modules, but technically could operate on the data since it is exposed as a regular JS ArrayBuffer. For this scenario, some amount of buffer copies are needed due to the nature of operating on three threads, so it doesn't seem like being able to create an ArrayBuffer on an existing native memory buffer would really change anything for our scenario. Since our scenario seems similar to your scenario (I might be missing something though), it's unclear to me how much it would help your scenario.

[mrousavy]

Hi @ryantrem thanks for the detailed answer!

To answer those questions;

You are getting some kind of RGB buffer from OS APIs (either as a texture or some kind of raw memory buffer)

Right

You get those buffers on some thread that is related to camera capture (not the JS thread)

Wrong, I am running a separate workletized JS-Runtime on the same Thread where I get those buffers. That's why I need to be able to re-use a single buffer, because I exactly know when it's no longer being used, everything's synchronous.

I don't want to create a new ArrayBuffer per frame (allocating 10MB at 240 FPS is not realistic), and I don't want to re-use a single ArrayBuffer instance because that's still one memcpy operation happening (not necessarily a problem because it is slow, but rather because it's duplicate memory - so 20 MB. What happened to being memory efficient?)

Doing any kind of per-pixel operation on a 10mb RGB buffer at 240hz (or even 60hz) on the JS side is impractical from a performance standpoint (I don't think you could even iterate over the elements of a 10mb ArrayBuffer at 240hz on the JS side, let alone do any processing on the data - JS is slow without JIT)

It might be, yes. But I'm mostly doing this for an experiment anyways, so I wanted to see how that would work out. Maybe some basic tasks work fine. Also I'm not doing this at 240 Hz, in a real world use case this will happen at 1 FPS up to 30 FPS. Also the user can then pass the ArrayBuffer to a native func anytime.

[neildhar]

Added in e6d887a.

[wcandillon]

@neildhar Thank you for this feature, it's super useful 🙏