add option to only use HEAD version of git when installing

[marcmo]

vim-plug is really fast when you update your plugins. but what can take a rather long time to do a fresh install (some plugins have a rather long git history).
but actually I'm not interested in the history.

e.g. I just installed YCM which is a rather huge repo:

Updated. Elapsed time: 170.097284 sec.
[============================================]
...

191M ./YouCompleteMe

if I just clone the HEAD version it with git:

> time git clone --depth=1 https://github.com/Valloric/YouCompleteMe.git
Cloning into 'YouCompleteMe'...
remote: Counting objects: 72, done.
remote: Compressing objects: 100% (67/67), done.
remote: Total 72 (delta 3), reused 41 (delta 0), pack-reused 0
Unpacking objects: 100% (72/72), done.
Checking connectivity... done.

real    0m2.319s
user    0m0.090s
sys 0m0.084s

compare that to the full repo:

> time git clone https://github.com/Valloric/YouCompleteMe.git
Cloning into 'YouCompleteMe'...
remote: Counting objects: 29994, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 29994 (delta 0), reused 0 (delta 0), pack-reused 29987
Receiving objects: 100% (29994/29994), 29.15 MiB | 576.00 KiB/s, done.
Resolving deltas: 100% (9690/9690), done.
Checking connectivity... done.

real    1m0.539s
user    0m2.410s
sys 0m2.568s

[vheon]

Duplicate of #112?

[junegunn]

Yep shallow clone and sparse checkout were suggested in #122. I still don't see the point of sparse checkout, but now I think shallow clone can be a good addition to vim-plug. I was reluctant at first because of some issue reports, for example this one: Shougo/neobundle.vim#81 (comment). But it looks like the problem is in git fetch --depth 10 and not in git clone --depth 1. I don't think vim-plug will ever have to fetch plugin with --depth option, so it's not something we have to worry about. And I personally had no issue with shallow clones. What do you guys think? If we can be sure that shallow clone does not cause any problem, we can probably do it as the default, instead of adding an option. Also, in order to support switching of branches we're going to put --no-single-branch option as well.

[starcraftman]

@junegunn So the proposal is make clone command use git clone --depth 1 --no-single-branch as default command. I don't see any issue, long as problems you mention only happened on fetch. The only hitch I see is that some devs might be annoyed at having to use git fetch --unshallow/--depth 100000 to convert a repo and get all history. Otherwise, the code changes should be minimal.

[junegunn]

long as problems you mention only happened on fetch

I have no hard evidence on this, but have had no issue with it and don't see how it could go wrong.

some devs might be annoyed at having to use git fetch --unshallow/--depth 100000 to convert a repo and get all history

That's a good point. Plugin developers will notice the change. But I don't want to bother the majority of the users with another option. Hmm.

Another issue is that --single-branch option was added in git 1.7.10 so vim-plug will fail on old systems, unless we optionally apply the option. The option was added in 2012 which actually isn't too long ago.

[vheon]

The only hitch I see is that some devs might be annoyed at having to use git fetch --unshallow/--depth 100000 to convert a repo and get all history

That was exactly what I was thinking.

That's a good point. Plugin developers will notice the change. But I don't want to bother the majority of the users with another option. Hmm.

Could we bother the minority of users who writes their own plugin with an option??

[junegunn]

Could we bother the minority of users who writes their own plugin with an option??

Good point. We probably should add a switch if we're really going to do this.
I'm currently testing shallow-clone branch by installing YCM, and it's still taking forever compared to the others. The plugin is truly the root of all evil.

[starcraftman]

@junegunn The problem with YCM is it also has a loads of submodules. Even if this gets pushed, those still take time. I noticed some even get cloned twice because some are dependencies of YCM & the separate ycmd server.The test @marcmo ran didn't use --recursive so it wasn't very realistic.

Regarding implementation of this, I'm in favour of an option. Perhaps:

function! s:clone_opt()
  return get(g:, 'plug_unshallow', 0) ? '' : '--depth 1'
        \ . s:git_version_requirement(1, 7, 10) ? ' --no-single-branch' : ''
endfunction

[vheon]

@starcraftman so the g:plug_unshallow would be a general option, right?? So I cannot use the shallow clone for every plug but the one I develop?

[junegunn]

@starcraftman @vheon
I ran into vim-plug timeout from YCM even after the patch, haha now the whole point of shallow clone is very questionable to me 😑

I cannot use the shallow clone for every plug but the one I develop?

An option for global shallowness requires less changes in code and is simpler to use, so I like the approach, but yeah, in the end we probably should make it per-plugin configurable. But one may argue that for completeness' sake we need both global and local options.

[vheon]

But one may argue that for completeness' sake we need both global and local options.

Yes, but after:

I ran into vim-plug timeout from YCM even after the patch, haha now the whole point of shallow clone is very questionable to me 😑

I don't see the real value of the feature itself.

[starcraftman]

@junegunn I think there are a few things to clarify here.

1. YCM is a big corner case, it has so many submodules that it will invariably take a long time. The smallest size I could get it down to is 172M with all submodules at depth 1.

2. I realized that git clone --recursive --depth 1 does not propagate the depth argument to submodules. Source. The command we'd need is available with git >= 1.8.4. git clone --depth 1 https://github.com/MyRepo.git ; cd MyRepo ; git submodule update --init --depth 1 --recursive

3. I made the appropriate changes on a fork of shallow-clone & ran an unscientific test with my own vimrc to see speedup. I red time off top of window & size with du -s /tmp/vim/plugged.

Git Branch	YCM Included	Time Taken	Final Size of plugged
shallow-clone	✔️	138s	211M
master	✔️	184s	246M
shallow-clone	✖️	14s	39M
master	✖️	29s	50m

4. So even without YCM there is a speedup benefit as you'd imagine.

5. Re timeout, I think that is a separate issue relating to our implementation. Timeout in that case occurs usually because no log was produced on a long download of submodule. This feature imo is solely about speed/disk space.

[junegunn]

@starcraftman Thanks for the write-up. The result is not radically different from what I'd imagined. My point was that YCM still takes too long even with the patch. As it takes longer than 2 minutes (it took much longer for me yesterday), users will "multi-task" in other tmux window or something instead of staring into the installation screen until completion. And YCM aside, it's arguable if the 15-second reduction only during initial installation really compensates the cognitive overhead by a new, insignificant option we probably have to add.

I realized that git clone --recursive --depth 1 does not propagate the depth argument to submodules.

Yeah, I also noticed that while test-installing YCM. Can it be a bug of git? It's sad that we'd need an extra check of s:git_version_requirement(1, 8, 4) because of that :(

Having said that, I'm still tempted to make shallow-cloning default, as every user will benefit from it to some degree. What I'd like to avoid is to add global/local pair of shallow option. Maybe we can just tell the plugin developers who want the full history to put { 'do': 'git fetch --unshallow; :' } to their plugins.

There are a number of options:

1. Shallow-clone by default, tell the users to git fetch --unshallow if they want
2. g:plug_shallow (default: 0)
3. g:plug_shallow (default: 1)
4. Plug ..., { 'shallow': 1 } (default: 0)
5. Plug ..., { 'shallow': 0 } (default: 1)
6. Plug ..., { 'shallow': 1 } (default: g:plug_shallow = 0)
7. Plug ..., { 'shallow': 0 } (default: g:plug_shallow = 1)

EDIT: 8. Leave it as it is.

[junegunn]

My vote is 3 :)
(Or start from 2, and flip the default in the future)

[vheon]

I would like option 5 but I don't want to pay the cost of parsing yet another option for every plug on startup for a feature only useful for Installation. So considering that git fetch --unshallow is possible when I need the whole history I'm ok with option 3.

5. Re timeout, I think that is a separate issue relating to our implementation. Timeout in that case occurs usually because no log was produced on a long download of submodule.

How is that possible?? I mean doesn't git produce some output when downloading the submodule??

[junegunn]

How is that possible?? I mean doesn't git produce some output when downloading the submodule??

Yeah, but it did happen to me from time to time. Maybe GitHub (or my internet connection) was extremely slow at the time. I'm running git clone --progress --recursive --depth 1 https://github.com/Valloric/YouCompleteMe.git right now, and observing 5~10KiB/s. It's really frustrating.

[vheon]

Yeah, but it did happen to me from time to time. Maybe GitHub (or my internet connection) was extremely slow at the time. I'm running git clone --progress --recursive --depth 1 https://github.com/Valloric/YouCompleteMe.git right now, and observing 5~10KiB/s. It's really frustrating.

Yeah, I get that too. 😭

[junegunn]

I wonder if they're interested in improving the installation experience of YCM. A compact fork of the project which downloads prebuilt binaries via CDN.

[junegunn]

FYI: I'm still cloning YCM. This is absurd.

[starcraftman]

Yeah, I also noticed that while test-installing YCM. Can it be a bug of git? It's sad that we'd need an extra check of s:git_version_requirement(1, 8, 4) because of that :(

I don't know, seems a bit odd from a user POV. Perhaps the git guys intended it like that for backwards compatibility since older versions did complete clones of submodules. I might go look on their tracker to verify.

I wonder if they're interested in improving the installation experience of YCM. A compact fork of the project which downloads prebuilt binaries via CDN.

Perhaps, so something like Valloric/YCMPrebuilt & maybe the branches represent your build like linux-x64-clang. Might work. Sure would beat all those submodules followed by long build. I think they'd need to make something similar to NeoVim's marvim travis bot that does daily builds on travis and uploads.

Regarding vote, I'm for 3. In my opinion, most users will only ever want this on, unless they are doing some debugging. We should note this change at top of readme, maybe under 'pros' so that users see the change go in effect.

FYI: I'm still cloning YCM. This is absurd.

Clearly evil. 😈

[starcraftman]

@junegunn Was reading up on submodules from that link and noticed something. Doesn't seem like you can do depth=1 on a repo & its submodules, unless I got the command wrong. I believe the problem has to do with way submodules refer to git history tree.

Example not working (no --recursive).
git clone --depth 1 --no-single-branch --progress https://github.com/Valloric/YouCompleteMe.git -b master YCM && cd YCM && git submodule update --init --depth 1

What I had on my fork. Due to --recursive works but submodules history isn't depth 1. Verify by looking at YCM/.git/modules/third_party.
git clone --depth 1 --no-single-branch --progress --recursive https://github.com/Valloric/YouCompleteMe.git -b master YCM && cd YCM && git submodule update --init --depth 1

So we can put depth=1 for all top repos, but not submodules. Might be a git limitation?
Example working without submodule invocation:
git clone --depth 1 --no-single-branch --progress --recursive https://github.com/Valloric/YouCompleteMe.git -b master YouCompleteMe.

YCM probably should stop using submodules at this point, just going to continue making install longer & longer due to ever increasing history of all projects involved.

You might want to further test/research this to confirm, but that's what I understand.

[vheon]

YCM probably should stop using submodules at this point

And how would you handle those required libraries?

[starcraftman]

@vheon Probably something along the lines of the prebuilt thing @junegunn suggested. To note, just the ycmd submodule has itself 9 submodules that take time. All just to build ycmd server. If there were some command like :YcmdUpgrade that pinged a cached build server that would considerably save space/time.

[junegunn]

@starcraftman Thanks for the investigation. Hmm, I'll also take a look at it.

@vheon YCM already has the install script, so I can imagine it shallow-cloning the prerequisites instead of depending on git submodules, though it may not be as elegant.

[vheon]

So with:

YCM probably should stop using submodules at this point

What you're saying is that: for speedup installation time YCM should look at other approaches instead of cloning the project like it does now. At first I thought that you were saying to stop using submodule altogether.

If there were some command like :YcmdUpgrade

That actually IMHO is not in the scope of YouCompleteMe which is merely a client for ycmd.

[starcraftman]

@vheon At the end of the day ycmd (and submodules) download & build is the longest bit. I'm sure somehow it could be cached & retrieved, but that'd involve work Valloric probably isn't interested in at this point. Maybe something for a fork to do/manage. Until then, only thing we can do to go faster is depth 1, but YCM will continue to be slow as molasses installing/building.

[junegunn]

Merged the PR. Now vim-plug will do shallow clone by default.

[junegunn]

Hmm, no. I realized that shallow clone cannot include tags so tag option will not work.
So what should we do?

• No shallow clone when tag option is set.
  • But what if PlugInstalled w/o tag, then PlugUpdate with a tag?
    • git fetch --unshallow when tag is set
• Make this opt-in only
• Forget about shallow clone

[marcmo]

@junegunn first of all thanks for paying attention to this issue! I still think it can be very valuable since setup times and download sizes might be an issue for someone (like me)
I see then YCM really sucks regarding the whole setup and installation. I was reluctant to use it myself but since I switched to neovim and thus cannot use Shougo's excellent neocomplete anymore so YCM seemed a valid option. Plus it seems to be very popular.
The owner of YCM (Valloric) seems to be rather difficult to work with so I don't think he has any interest in making things easier for his users...well...anyway. Shougo will for sure come up with a valid neovim solution:)

about the tag problem: this is a valid point! :( I'd be in favor of disabling the shallow clone in that case.
then everything works as before for that case and we still get some benefit for the other plugins.

[junegunn]

Merged #221.

[elyscape]

Maybe this should also do shallow clones/updates for the submodules? It doesn't appear to do so at present.

[starcraftman]

@elyscape I investigated that with YCM, didn't seem so easy. See my post under 'clearly evil'. If I got something wrong I'm open to correction or better info. Ultimately, seems to me submodules aren't meant for small footprint download due to way they worked, else I imagine git devs would have simply propagated depth 1 to them.

[elyscape]

Ahh, whoops. Missed that somehow.

[ryneeverett]

An additional pro of shallow submodule clones is that it would avoid fetching requests' invalid git history, which breaks YCM installation when fsckobjects = true.

[ZSaberLv0]

@junegunn for me, --single-branch would make the git repo even smaller
would you consider to add options for --single-branch or --no-single-branch ?
for most case, i dont think plugin users want to care about the plugin's branches, only HEAD should be done

[junegunn]

@ZSaberLv0 Feel free to fork the repo and use your own version. vim-plug has been very stable, and you probably don't have to worry about future changes.