Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a helper to generate Blob (lazily read) from a file path #68

Closed
coyotte508 opened this issue Mar 8, 2023 · 3 comments · Fixed by #104
Closed

Add a helper to generate Blob (lazily read) from a file path #68

coyotte508 opened this issue Mar 8, 2023 · 3 comments · Fixed by #104
Labels
hub @huggingface/hub related

Comments

@coyotte508
Copy link
Member

coyotte508 commented Mar 8, 2023

Or wait on nodejs/node#45188

This would help for the commit calls, when uploading very large files, to avoid loading the whole file in memory.

Would help with #62

Technical PR.


Given that it's only needed for Deno & Node (and node should have a native solution soon), it should be an internal helper. We want to be able to remove it at some point.

@coyotte508 coyotte508 added the hub @huggingface/hub related label Mar 8, 2023
@Aschen
Copy link
Contributor

Aschen commented Mar 17, 2023

I started to look into this and if I understood well the problem, I'm thinking about a solution using Filehandle.read but it will need some change in the code because then slice method need to be async in order to read the desired slice from the file and returning it in a Blob instance.

Also about the usage Blob.length here and here, isn't it an error since it does not exists on the Blob API?

@coyotte508
Copy link
Member Author

Also about the usage Blob.length here and here, isn't it an error since it does not exists on the Blob API?

Thanks! I added a bug fix: b7934a5

I started to look into this and if I understood well the problem, I'm thinking about a solution using Filehandle.read but it will need some change in the code because then slice method need to be async in order to read the desired slice from the file and returning it in a Blob instance.

It should be fine to only do the Filehandle calls when .arrayBuffer() / .text() / .stream() is called. As for .size, if we pass the info to the new blob when creating it, it should be fine as well.

@Aschen
Copy link
Contributor

Aschen commented Mar 17, 2023

I suggested a LazyBlob implementation here #104

Feel free to give me any suggestion

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hub @huggingface/hub related
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants