GitDownloadStrategy change breaks HEAD fetch for Julia

[staticfloat]

I'm (still) working on the Julia formula, and I've run into an interesting problem:

Apparently, a recent commit changed how git URLs are fetched, and there's something special about the Julia repository, (https://github.com/JuliaLang/julia.git) because it gives the following error when trying to install:

$ brew install -vd --HEAD julia
==> Installing julia
==> Cloning https://github.com/JuliaLang/julia.git
Updating /Library/Caches/Homebrew/julia--git
git config remote.origin.url https://github.com/JuliaLang/julia.git
git config remote.origin.fetch +refs/heads/master:refs/remotes/origin/master
git fetch origin --depth 1
fatal: git fetch-pack: expected shallow list
fatal: The remote end hung up unexpectedly
Error: Failure while executing: git fetch origin --depth 1
/Users/<username>/.homebrew/Library/Homebrew/utils.rb:104:in `safe_system'
/Users/<username>/.homebrew/Library/Homebrew/download_strategy.rb:27:in `quiet_safe_system'
/Users/<username>/.homebrew/Library/Homebrew/download_strategy.rb:364:in `fetch'
....

I googled around a bit, but can't really find much information on this error. @jacknagel, any idea why your recent changes would cause this, and what I can do to fix it? Could it be an upstream issue? (I tried a few other HEAD formulae, but they all worked)

$ git --version
git version 1.7.10.1

[jacknagel]

Link to the copy of the formula you're using?

[staticfloat]

Of course, sorry: https://github.com/staticfloat/homebrew-science/blob/0f519051c358926ea19c80ed0224ba70bb37848e/julia.rb (The formula isn't polished yet, still working on some dependencies, etc...)

Note that it depends on a bunch of stuff that has not yet been accepted into homebrew/science, but if you comment out the dependencies, it should fail on the download step.

[jacknagel]

Works fine here.

[jacknagel]

Initial fetch, and a second install after the initial clone: https://gist.github.com/2583606

[staticfloat]

I deleted the cached git repository, and it works now (both initial clone and update). Something must have been screwed up with my local git repo.

[Sharpie]

Looks like this is happening when git 1.7.7.5 from XCode is used to do the initial clone:

➜  Homebrew  /usr/bin/git clone --depth 1 https://github.com/znc/znc.git /Users/sharpie/Library/Caches/Homebrew/znc--git
Cloning into '/Users/sharpie/Library/Caches/Homebrew/znc--git'...
remote: Counting objects: 2706, done.
remote: Compressing objects: 100% (1818/1818), done.
remote: Total 2706 (delta 2090), reused 1213 (delta 810)
Receiving objects: 100% (2706/2706), 1.55 MiB | 304 KiB/s, done.
Resolving deltas: 100% (2090/2090), done.
➜  Homebrew  cd znc--git 
➜  znc--git git:(master) /usr/bin/git fetch origin --depth 1
fatal: git fetch-pack: expected shallow list
fatal: The remote end hung up unexpectedly
➜  znc--git git:(master) /usr/local/bin/git fetch origin --depth 1
fatal: git fetch-pack: expected shallow list
fatal: The remote end hung up unexpectedly


➜  znc--git git:(master) cd ..
➜  Homebrew  rm -rf znc--git 
➜  Homebrew  /usr/local/bin/git clone --depth 1 https://github.com/znc/znc.git /Users/sharpie/Library/Caches/Homebrew/znc--git
Cloning into '/Users/sharpie/Library/Caches/Homebrew/znc--git'...
remote: Counting objects: 258, done.
remote: Compressing objects: 100% (232/232), done.
remote: Total 258 (delta 16), reused 98 (delta 4)
Receiving objects: 100% (258/258), 660.56 KiB | 286 KiB/s, done.
Resolving deltas: 100% (16/16), done.
➜  Homebrew  cd znc--git 
➜  znc--git git:(master) /usr/local/bin/git fetch origin --depth 1
remote: Counting objects: 254, done.
remote: Compressing objects: 100% (226/226), done.
remote: Total 254 (delta 14), reused 86 (delta 4)
Receiving objects: 100% (254/254), 628.25 KiB | 271 KiB/s, done.
Resolving deltas: 100% (14/14), done.
From https://github.com/znc/znc
 * [new branch]      0.204-bugfix -> origin/0.204-bugfix
➜  znc--git git:(master) /usr/local/bin/git fetch origin --depth 1
remote: Total 0 (delta 0), reused 0 (delta 0)
➜  znc--git git:(master) /usr/bin/git fetch origin --depth 1      remote: Total 0 (delta 0), reused 0 (delta 0)
➜  znc--git git:(master) /usr/local/bin/git fetch origin --depth 1
remote: Total 0 (delta 0), reused 0 (delta 0)

[jacknagel]

I just built 1.7.7.5 from source and I can reproduce it there. I'll investigate it further tomorrow.

[pivotal-casebook]

We are seeing the same thing, even with a newer version of Git installed via Homebrew.

perry:~  $ git --version
git version 1.7.10.1
perry:~  $ brew install macvim --HEAD
Warning: You have Xcode 4.2.1, which is outdated.
Please install Xcode 4.3.
==> Cloning https://github.com/b4winckler/macvim.git
Updating /Users/pivotal/Library/Caches/Homebrew/macvim--git
fatal: git fetch-pack: expected shallow list
fatal: The remote end hung up unexpectedly
Error: Failure while executing: git fetch -q origin --depth 1

[Sharpie]

It seems like if the older version did the initial clone, then even a newer version will fail to update---can't tell from your example which version did the initial clone.

Could you try deleting the MacVim repo from brew --cache and trying again.

[jacknagel]

We can drop the --depth from the git fetch invocation, though that is kind of unfortunate.

[Sharpie]

One thing we should check is to see if this happens with non-GitHub URLs---just on the off chance that GitHub is doing something wrong.

[jacknagel]

Looking at the git code, it appears that error can only be triggered over HTTP. So I would try some other smart HTTP hosts (I know llvm HEAD supports shallow clones; to get Homebrew to use them, though, one has to make GitDownloadStrategy#host_supports_depth? return true).

[jacknagel]

I can reproduce this on four different git versions, including the latest, but I think this may actually boil down to a misuse of the --depth option to git fetch.

It is supposed to "deepen" the clone by <n> commits, and while the documentation isn't 100% clear, it works for me if I increment the number each time I invoke fetch. Luckily, it also appears that as long as the fetch refspec is set correctly, we don't actually need to specify a depth to preserve the shallowness of the clone.

[jacknagel]

Should be fixed by 8a1c2ec.

[sheerun]

I attach here repo with similar problem. Try to call git fetch --depth=10, it fails. Then try git fetch --depth=20, it succeeds. It was cloned with NeoBundle. About 10% of cloned packages have this problem. http://www.speedyshare.com/n3MqB/ack.tar.gz

This fails:

git clone https://github.com/mileszs/ack.vim --depth=1 && cd ack.vim && 
  git fetch --depth=10 && git fetch --depth=10

This succeeds:

git clone https://github.com/mileszs/apidock.vim.git --depth=1 && cd apidock.vim && 
  git fetch --depth=10 && git fetch --depth=10