Support streaming reading of large files

[aredridel]

Especially for an unpacked repo of large files (shallow clone of multi-gb dataset for example), it'd be REALLY nice to have a streaming interface to read data with.

[creationix]

I'm thinking this only makes sense for blobs. Maybe we could add a new type to loadAs and saveAs called "stream". Then in the db interface we would add optional getStream and setStream primitives to the interface. For backends that are unable to implement streaming, js-git will just use the non-streaming interface and buffer internally.

What do you think?

[aredridel]

Seems reasonable; I'd love streaming for trees too but it's just for consistency in that case.

loadAs is a weird interface though -- you have to know the type ahead of time, and since trees can contain both trees and blobs, that's weird.

[creationix]

Interesting. If you have other API ideas, I'm open. I will mention that
with some backends like js-github you are required to know what kind of
object you're loading before loading it. I get around this in js-github by
caching hash <-> type information wherever I can. (If you have a hash, you
got it from somewhere and probably already know the type from where you got
it).

On Fri, Dec 13, 2013 at 3:44 PM, Aria Stewart [email protected]:

Seems reasonable; I'd love streaming for trees too but it's just for
consistency in that case.

loadAs is a weird interface though -- you have to know the type ahead of
time, and since trees can contain both trees and blobs, that's weird.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/83#issuecomment-30546145
.

[aredridel]

Oh, interesting -- since github doesn't expose raw git in a browser-accessible way, eh?

[creationix]

Because it uses this api: http://developer.github.com/v3/git/ Which
requires the type in the request.

On Mon, Dec 16, 2013 at 3:10 PM, Aria Stewart [email protected]:

Oh, interesting -- since github doesn't expose raw git in a
browser-accessible way, eh?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/83#issuecomment-30700923
.

[aredridel]

Oh, I'd forgotten that trees internal representation does include the type of the next nodes. That makes it less heinous.