Adding Chunk File Upload Support to the library

[PauloPeres]

Fixing Issues #880 and #435
With just a few tweeks on the implemations you can use the chunk file upload method.
4 new Parameters

chunkSize: (1024*1024)*2, // Size of the chunks in bytes
currentChunkParam: 'current_chunk', // Current chunk key parameter to be sent to your endpoint
totalChunkParam: 'total_chunks',  //Total Chunk key parameter to be sent to your endupoint
chunkMethod: 'PUT', // Chunk Method Default Put for update of file

and 1 new callback to change the link of that specifc item on the request

this.uploader.onCompleteChunk = (item,response,status,headers)=>{
            response = JSON.parse(response);
            if(response['id']){
                item.url = new_link+response['id']+'/';
            }
        }

[codecov]

Codecov Report

Merging #977 into development will decrease coverage by 2.26%.
The diff coverage is 24.4%.

@@               Coverage Diff               @@
##           development     #977      +/-   ##
===============================================
- Coverage        34.72%   32.46%   -2.27%     
===============================================
  Files               10       11       +1     
  Lines              550      807     +257     
  Branches            80      106      +26     
===============================================
+ Hits               191      262      +71     
- Misses             341      527     +186     
  Partials            18       18

Impacted Files	Coverage Δ
src/file-upload/file-uploader.class.ts	24.36% <17.54%> (+0.17%)	⬆️
src/file-upload/file-chunk.class.ts	27.53% <27.53%> (ø)
src/file-upload/file-item.class.ts	24.82% <32.55%> (+3.39%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fb48f44...45e1812. Read the comment docs.

[mbaljeetsingh]

How can I use this to chunk file upload support to Azure storage, Is there any documentation available?

[PauloPeres]

@mbaljeetsingh Hi man as you requested i added a little bit more examples on the ReadMe of my Branch
https://github.com/PauloPeres/ng2-file-upload
I'm currently using with Azure Blob-Storage as well, but i don't recommend using the Angular Code directly to connect with your Blob Storage because this Might Expose your credentials to the Client side.
On your Server side create the logic to make the bridge between the received chunk and the Blob-Storage.
Hope this helps!

[mbaljeetsingh]

Thanks I'll take a look. I'm actually getting sas token based on the logged in user credentials. So, it seems secure 😉

[mbaljeetsingh]

@PauloPeres Can you push your code to separate npm package? Otherwise I think I have to copy paste the updated files to ng2-file-upload node_modules folder.

Is there's a better solution?

[PauloPeres]

@mbaljeetsingh Hi Man....
So i did not wanted to push to the NPM becuase i woudl like to see someone whos taking care of the ng2-file-upload to put it to work.
in the meantine what i did with my projects is:
added the folder src in my project and i'm using the typescript package this way i can change and improve as i go.
If i dont get response from de delovelopers for too long will put on the NPM :D

[mbaljeetsingh]

Ok, I'll give it a try like this

[mbaljeetsingh]

@PauloPeres Got the import working for your code.
Here's my working code without Chunk support, I'm using dynamic URL for each file.

videoId = this.guid();
cameraId = this.guid();
sasToken: any;
files: {fileName:string}[] = [];

// Get SAS Token
this.accountService.getSasToken().subscribe(data => {
      this.sasToken = data;
});
this.uploader = new FileUploader({
      itemAlias: "files",
      method: "PUT",
      autoUpload: true,
      headers: [{ name: "x-ms-blob-type", value: "BlockBlob" }]
});    

this.uploader.onAfterAddingFile = file => {
      file.withCredentials = false;
 };

this.uploader.onAfterAddingAll = (files: FileItem[]) => {
      files.forEach(fileItem => {
        fileItem.url = `${environment.blob_service_endpoint}/${environment.container_name}/${this.videoId}/${this.cameraId}/${fileItem.file.name}${this.sasToken.token}`
      });
    };

    // Track on complete item
    this.uploader.onCompleteItem = (
      item: any,
      response: any,
      status: any,
      headers: any
    ) => {
      this.files.push({fileName: item.file.name});
      // console.log(this.files);
    };

    this.uploader.onCompleteAll = (): any => {
      // Store Video MetaData
      this.uploadService.updateVideoMetadata(this.videoId, this.cameraId, this.files).subscribe(data => {
       console.log("metadata save");
      });
    };

As I'm using SAS token, I'm not sure how should I set the item URL in the case of adding chunk support.
I'm tring to update your code as follows,

this.uploader = new FileUploader({
    // url: URL,
    disableMultipart : false,
    isHTML5: true,
    chunkSize: (1024*1024), // 2MB
    currentChunkParam: 'current_chunk',
    totalChunkParam: 'total_chunks',
    chunkMethod: 'PUT',
    autoUpload: true
  });
  this.uploader.onBeforeUploadItem = (item) => {
      // If you use credentials this might help you with the "Access-Control-Allow-Origin" error
      item.withCredentials = false;
  };

  this.uploader.onCompleteChunk = (item, response, status, headers) => {
   // Stuck Here
    response = JSON.parse(response);
    console.log(response);
    
    }
  };
  this.uploader.onErrorItem = (item, response, status, headers) => {
     // Treat the error on the upload
     // On the chunk method we try to upload a chunk for 10 times before triggering this error
  };
  this.uploader.onRemoveItem = (item) => {
     // Treat the file removal from the server
  };

As I'm not updating URL in the this.uploader.onAfterAddingAll , I'm getting the following error,

So, I'm stuck. Can you suggest, how should I go about If I need to use SAS token?

[PauloPeres]

@mbaljeetsingh use the onBeforeUploadItem and set item.url as the url you need for that specif item to be posting the chunks

[mbaljeetsingh]

@PauloPeres Updated the URL in onBeforeUploadItem, But when the upload start it is doing post request on the URL, so getting the error (The resource doesn't support specified Http Verb.)

Shouldn't it send the put request?

[mbaljeetsingh]

@PauloPeres I updated the uploader object with method and headers,

this.uploader = new FileUploader({
    // url: URL,
    disableMultipart : false,
    isHTML5: true,
    chunkSize: (1024*1024), // 2MB
    currentChunkParam: 'current_chunk',
    totalChunkParam: 'total_chunks',
    chunkMethod: 'PUT',
    autoUpload: true,
    method: "PUT",
    headers: [{ name: "x-ms-blob-type", value: "BlockBlob" }]
  });

Now it seems to start uploading, but I'm not getting any response back in onCompleteChunk

this.uploader.onCompleteChunk = (item, response, status, headers) => {
    // response = JSON.parse(response);
    console.log(response);
  };

[PauloPeres]

So it's getting error on the uploadcheck the uploader.onErrorItem callback to see if the first chunk is going or what kind of error you are getting from it. i don't know who you should send data to the blob storage letting it know what chunk number you are sending,
This link has documentation for it.
The first post should be POST with information about the chunks
https://docs.microsoft.com/en-us/rest/api/storageservices/reliable-uploads-to-blob-storage-via-an-html5-control

[mbaljeetsingh]

@PauloPeres Can you share how you structured your API, I may ask the developer to make modifications accordingly?

I'm getting the following response on both onErrorItem & onCompleteChunk,

<?xml version="1.0" encoding="utf-8"?><Error><Code>MissingRequiredHeader</Code><Message>An HTTP header that's mandatory for this request is not specified.
RequestId:22024d56-001e-00db-6ddb-ba397d000000
Time:2018-03-13T14:57:19.9743796Z</Message><HeaderName>x-ms-blob-type</HeaderName></Error>

The problem I think is, when we use SAS token, It is expecting a put request to the url similar to the following,

“https://<storage_account_name>.blob.core.windows.net/<container_name>/“+file.name+“sas_content“

We also need to pass the header,

headers: [{ name: "x-ms-blob-type", value: "BlockBlob" }]

Console

[PauloPeres]

@mbaljeetsingh Hello Again : )
The API we work is built in Pythonwe used the Azure Blobl SDK and used teh AppendBlobService To append the chunks
On the Logic we first create the Blob Storage give it the name and the start content (first chunk), after that we have a Blob Storage Reference to work on the next chunk requests.
Then every new chunk we user the AppendBlobService to append the data

You should be able to copy the same logic to TypeScript

Theres useful information on the Node Module from Microsoft
Create the stream for a new file http://azure.github.io/azure-storage-node/FileService.html#createWriteStreamToNewFile
Create the stream for an existing file http://azure.github.io/azure-storage-node/FileService.html#createWriteStreamToExistingFile__anchor

[bvercelli99]

@PauloPeres Any chance you can publish this to your own npm project? It appears the devs behind ng2-file-upload haven't made any changes or been active on this project in a long time. I would really like to use the chunk file upload if possible.

[ph3n0m666]

definitely would like to see this as an npm project

[PauloPeres]

Hi @ph3n0m666 ad @bvercelli99 Sorry for taking to long.
I'm still working on the NPM Package, it's my first time so i'm losing my virginity.

On the path of doing so, i updated the version of the Package to use Angular 6, and Instead of using the links, use a Service, this way we can have more control over the functions of the upload.

Will be trying to send that NPM Package ASAP! It's a little bit complicated because of the Day-to-Day Stuff :D

[sebastiangug]

@PauloPeres Just came across this on my search for an easier solution to uploading to Azure Blob Storage, I'm using the preview to do it with Azure Active Directory tokens which should be just great to use on the client side.

Not being able to find this on npm made me a bit weary of using it, is this still something that's going on or has the main ng2-file-upload picked up on the chunk feature?

[MgSam]

Was this ever moved to a separate repo and published on npm?

[PauloPeres]

@MgSam never done it... Let me update to Angular 7 and will deploy it to NPM

[jbjhjm]

Looks promising. Tried to npm install it from git repo and build it but there were some issues, so I guess ones best bet is to copy changes manually or wait for the official package to be released. Thanks @PauloPeres !

[chioai1309]

Hi @PauloPeres,

Is there any possibility to publish this feature to ng2-file-upload? This really helpful and match exactly what I'm looking for.

[mehmetranas]

@PauloPeres is there any way to abort a chunk upload process? Because item.cancel() not workinkg in chunk model.

[PauloPeres]

The cancel should work. After the first chevuk is accepted, só dont allow your user to delete a file before having the first chunk Em qui, 12 de mar de 2020 12:27, Mehmet <[email protected]> escreveu:

…

@PauloPeres <https://github.com/PauloPeres> is there any way to abort an chunk upload? Because item.cancel() not workinkg in chunk model. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#977 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7CFFSNUYXE7ZEC5SGQWPDRHD5NZANCNFSM4EQC5GMQ> .

[mehmetranas]

@PauloPeres i cancel process in onCompleteChunk method, there is not any fail but upload process go on, i think after
onCompleteChunk = (item:FileItem, response, status, headers) => item.cancel()
upload process should abort but it doesn't.

[GopalZadafiya]

@PauloPeres Is commit 5a84b9e#diff-b8aa3b862dc83fee3efcdc29e54d959b valid ? I want to handle failed chunk upload

[PauloPeres]

sorry for the delay i'm trying to work that on a separated npm package but for now the latests commits are in the : https://github.com/myog-io/ngx-chunk-file-upload Kind Regards Paulo Peres Jr Phone: +1 (818) 279-3840 (tel:+1%20(818)%20279-3840) WhatsApp: +55 14 98225-0396 (https://api.whatsapp.com/send?phone=5514982250396)

…

On Apr 9 2020, at 5:47 am, Gopal Zadafiya ***@***.***> wrote: @PauloPeres (https://github.com/PauloPeres) does the commit 5a84b9e#diff-b8aa3b862dc83fee3efcdc29e54d959b (5a84b9e#diff-b8aa3b862dc83fee3efcdc29e54d959b) valid ? I want to handle failed chunk upload — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub (#977 (comment)), or unsubscribe (https://github.com/notifications/unsubscribe-auth/AA7CFFUNCVOYMH6FZJGCF2LRLWDTTANCNFSM4EQC5GMQ).