Feature request: allow specifying the size of the chunks provided by File#createReadStream()

[Dinduks]

Hello,

At the moment, the chunks returned by File#createReadStream() method are 16kb long.
I believe that's Node's default highWaterMark value.

The issue with this is that it consumes too much memory to read a big file.

For a 250mb file, that's 16k chunks, and it consumes 2GB of memory before the process crashes.

If you are okay with it, I'd like to submit a PR to specify the chunks size.

I will look into createReadStream() code tonight and see what I can do.

Meanwhile, any help or guidance is welcome.

Thanks,

Samy

[AVaksman]

@Dinduks Thank you for reporting this.
Could you please give more details, share your environment details

Environment details

• OS:
• Node.js version:
• npm version:
• @google-cloud/storage version:
• related modules versions:

Please share a code snippet that causes the issue.

I've tried to reproduce in a couple of ways (400mb):

• file.download into file/buffer
• file.createReadStream into file/buffer
  but unsuccessful

[Dinduks]

@AVaksman Hi Alex.

Indeed, there is no issue with @google-cloud/storage itself.

The memory issues come from the thousands of operations inside my .on('data', () => {}) on the stream created by createReadStream.

I think being able to grab chunks bigger than 16kb would solve this problem by reducing the number of operations inside .on('data', () => {}).

Meanwhile, I'll try to optimize those operations.

I could provide the sample if you think it's relevant.

Also:

• OS is Arch Linux kernel 5.2.8
• Node version is v11.15.0
• npm version is 6.10.3
• GS version is 3.3.0 or 3.2.1
• Used lib inside .on('data', () => {}) is https://www.npmjs.com/package/stream-buffers.

[jkwlui]

highWaterMark does not dictate the size of the chunk:

nodejs/node#8855

[Dinduks]

Ok @jkwlui.
Is there anything you suggest to increase the payload of the data provided by this library?

[jkwlui]

Would you mind providing us with a simple reproduction code that we can try and replicate? You mentioned using stream-buffers inside the event callback, so if you could provide an example of that it'd be great, thanks!

[stephenplusplus]

@Dinduks please feel free to provide more information, but I believe this is an issue to solve at the application level as opposed to our level in the library. You could look into using something like throttle, or otherwise queueing the operations you're running in the .on() handler in steps. You can also use the start and end options when reading a file to only handle chunks of the file at once.

I'm going to close the issue, but I'm happy to re-open if there's more we should do on our side.