Node.js stream support missing in LargFileUploadTask

[naugtur]

Bug Report

Prerequisites

• [ x ] Can you reproduce the problem?
• [ x ] Are you running the latest version?
• [ x ] Are you reporting to the correct repository?
• [ x ] Did you perform a cursory search?
  yes, I found a gist that suggested reading entire file into memory for each chunk splicing :)

Description

Attempting to pass a Stream into LargeFileUploadTask as file content is failing with

this.file.content.slice is not a function

const fileObject = {
    size,
    content: fileStream,
    name: targetFileName,
  };
  const uploadTask = await new LargeFileUploadTask(
    graphClient,
    fileObject,
    uploadSession
  );
  const response = await uploadTask.upload()

Steps to Reproduce

1. Take the example https://github.com/microsoftgraph/msgraph-sdk-javascript/blob/HEAD/docs/tasks/LargeFileUploadTask.md
2. use var readStream = fs.createReadStream(filename); to get a file stream and pass that in as content

Expected behavior: [What you expected to happen]

Stream flows and gets accumulated into chunks that get uploaded

Actual behavior: [What actually happened]

Error stemming from the assumption content is a Buffer

Additional Context

Streams allow for a container limited to 100MB of RM to upload a multi GB file successfully.
LargFileUploadTask is where it breaks.

Users of the SDK can't afford to load the entire file into RAM if they're building a multi tenant web app.

I'm willing to contribute a solution if I get someone from the maintainers to help me out (I have little experience using TypeScript and typing a stream is difficult)

Usage Information

SDK Version - [SDK version you are using]

• [ x ] Node (Check, if using Node version of SDK)

Node Version - [The version of Node you are using]

10+ (I used multiple)

AB#7546

[naugtur]

Here's what I'd like to do:

1. If it's possible to send entire upload to the session in one chunk (as discussed on Twitter) I'd like to expose a method to do that, which would consume anything the http client can put into body (string, blob, buffer, formdata, but most of all Stream)
2. If the file content passed to chunk upload helper is a stream:

• prepare two buffers (pooling instead of garbage collection 🎉)
• consume the stream
• collect it in a buffer as it flows
• cut off at the chunk size and continue streaming to the other buffer
• take the full buffer and send it as a chunk
• apply backpressure to the source stream while waiting for the upload request to process; keep the full buffer handy for retries in case they're needed. Discard (overwrite with another chunk in fact, because pooling) only on success

to implement backpressure a few tricky details in a Writable implementation would need to happen.

I'd build it in JS and then need help converting to TS
Also, testing would probably have to be end-to-end, because it's easy to get invalid results if you start mocking stream consumption instead of actually sending it over the network.

Action items:
Please let me know where to look for docs or code explaining (1)
Please let me know if you're interested in working on 2 and someone can allocate a few hours to help me out.

[MIchaelMainer]

@naugtur We'd be happy to have you contribute here. @nikithauc will work with you on this.

[naugtur]

Ok. Allocating time next week!

[MIchaelMainer]

Additional reference PRs:

https://github.com/microsoftgraph/msgraph-sdk-dotnet/pulls?q=is%3Apr+author%3Aandrueastman+large+is%3Aclosed
microsoftgraph/msgraph-sdk-dotnet-core#65

[naugtur]

Hi @nikithauc
Could you share info or docs about the option to send a single chunk to the upload session?

Darrel Miller mentioned that on Twitter.

Thanks for opening the issue. We should definitely investigate how to enable streaming bytes into range requests. We should also investigate if just doing a plain transfer-encoding: chunked request would also work to avoid range requests.

I'd like to start with proving I can make it work.

[nikithauc]

Hi @naugtur,

Currently it is not possible to send a single chunk in the upload session and the way to send a stream is through range requests.

The PRs mentioned above are an example of data being sent in ranges.

More references - Upload Chunk Request Implementation in dotnet, LargeFileUploadTask file in dotnet .

I generally use the references from other Graph SDKs so that the design of JS SDK implementation is in sync with the others.
Please let me know if you have more questions.

Regards,
Nikitha

[naugtur]

I can confirm the uploadSession demands all details about ranges.

Sending
Content-Range: 0-${N-1}/${N}
works for a single chunk upload of a file larger than 4MB though.
I tried the request directly, not via UploadTask.

Then I also uploaded 203MB powerpoint file as one chunk and it worked fine.
Any thoughts on that?
The only limitation on chunk size was a popular JS library that's a dependency in many http clients, but I switched it off and the API itself gobbled up 200MB and crumbs.

@nikithauc Can you ask someone internally and see if they're surprised I uploaded 200MB?

[MIchaelMainer]

uploaded 203MB... as one chunk and it worked fine.

I'm surprised as the max per documentation is 60MB.

You can upload the entire file, or split the file into multiple byte ranges, as long as the maximum bytes in any given request is less than 60 MiB.

@ificator do we need to update the documentation on this one?

[naugtur]

I'm happy with how it works but if it is unintended, please let me know and I'll avoid abusing it :)

I uploaded the file into a SharePoint site, if that changes anything.

Planning to work on converting stream to chunks anyway.

[baywet]

I've been working on this on the Java SDK recently. The upload services behave a bit differently between OneDrive consumer (live account) OneDrive for business, SharePoint and file attachments on Outlook. It worth testing all these scenarios to catch eventual issues.

[nikithauc]

@MIchaelMainer -> Came across this Stack Overflow post. The post mentions a download request but could it be the related to why a 203 MB chunk got uploaded as a single request?

[nikithauc]

@naugtur I know that the discussion for this issue is in the initial stage but as we move to the implementation design phase, I would like to work with you on the design part of this issue so that we have it in sync with the rest of MS graph SDKs .

Thanks,
Nikitha