Pull / push

[kmalakoff]

We've discussed a little bit about the challenges of interfacing with remote repositories (#122 and other issues).

I've noticed that this repository implemented clone using js-git utilities so it could be good to study.

Putting the protocol issues and conflicts / merging aside for a moment, I was wondering how one might go about implementing push and pull? for example, is there is sample code (even partially complete)? if not, what js-git utils would one use and how? Are there js-git utils that are missing and would need to be written?

Just trying to scope the problem and hoping for some pointers to see how to move this forward.

[creationix]

You'll want to have the protocol docs handy for sure. https://github.com/git/git/blob/master/Documentation/technical/pack-protocol.txt

https://gist.github.com/schacon/6092633

Part of fetch (the main part) is parsing a pack file and streaming the objects to the js-git data store. If I remember correctly, this needs raw loading and saving objects (used to apply delta objects). I've written code for clone a few times in a few different places. I even once had a node command-line tool that would clone using node + js, it's published to npm at https://www.npmjs.com/package/jsgit and you can probably find the code by installing it. (I'm pretty sure it's for an older incarnation of js-git though).

The only parts of js-git that are truly useful and re-usable are the codecs. Parsing and writing git objects, applying binary deltas given two buffers, streaming parsing of pack-files, etc. Feel free to change the rest to whatever works best.

[creationix]

At one point, I had a demo working that could clone over XHR (as long as you installed a browser extension to get around the cross-domain issue Github refuses to fix).

https://github.com/creationix/clone-test

https://github.com/creationix/corson

[kmalakoff]

@creationix thank you for the resources. I'll go through them and try to come up to speed on what is involved...famous last words!

Unless I'm mistaken, it looks like most of these example are cloning. Are there examples of push or pull as well? (I'm assuming clone is the easiest case since it is with a clean slate, right?) I'm just starting from zero knowledge on pack files so any code or tests to help me figure out some of the details on manage these probably complex operations will help me get up the learning curve fast. If not, that's fine too!

[creationix]

Clone and pull are the same at the protocol level (they are both fetch), but what you do locally with the refs is what makes them different. Push is a similar protocol, but I never implemented it because I never had a secure channel from the browser to github. Cloud hosted proxies work* for cloning public stuff, but nor for pushing changes because then the proxy can man-in-the-middle and get your credentials.

*work - as is it technically works, but it's a pain to host a proxy just to read a public resource.

[kmalakoff]

Thank you for clarifying.

When I have some time, I'll study something like libgit2 or the git source code to figure out what to do for pull and push - it is easiest with some working code as a starting place. Big learning curve ahead!

[creationix]

If working code is easier for you, so be it, but git tends to use a lot of mem-mapping that isn't possible in JavaScript. I personally had better luck just reading the protocol spec and then implementing from scratch, but then again, I'm not that good at reading other people's code yet.

[kmalakoff]

I just find that seeing the cases that need to be dealt with and how is helpful which is why I opened this issue in the first place! That said, I haven't looked at the code yet (but did develop in C/C++ for a few years) so it might end up being a blind alley...

I'll definitely end up doing a bit of both! Thank you for the advice.

FYI: I've actually been quickly / hackily using js-git as a driver layer for the nodegit API for this reason. I'm hoping that the process will teach me enough about git and provide enough working code to then refactor it into a higher level API. It's been a bit time-consuming, but very helpful for learning at least.