v0.5 Pkg operations through proxy failed

[yangguang760]

@staticfloat

As mentioned in the issue below
JuliaLang/julia#18613

Plz update the libgit2 build in the PPA to include curl support.

[staticfloat]

Thanks for the report @yangguang760, we will try to fix this quickly for you.

@tkelman we are building libgit2 with USE_SYSTEM_CURL=1. What can I do to try and trigger this on my local machine?

[tkelman]

I don't know exactly how to set up proxy testing. @ViralBShah @mdpradeep what did we do for testing JuliaLang/julia#17783 ?

[mdpradeep]

We set up a proxy server (Charlie) and made the following config:

[url "https://github.com/"]
    insteadOf = git://github.com/
[https]
    proxy = http://10.128.0.4:8888
[http]
    proxy = http://10.128.0.4:8888

[ViralBShah]

@mdpradeep Isn't that some private machine? Not sure if that helps here.

You can use ssh -v -D 2222 remote to create a proxy server that will start a socks5 proxy server listening on localhost:2222. The -v will make sure that connections get printed.

[staticfloat]

I backported a newer version of libcurl-dev (7.47), but that doesn't seem to have solved it. From what I can tell, the ubuntu Julia completely ignores ~/.gitconfig and just tries to connect directly every time. Any ideas on when/how libgit2 looks at ~/.gitconfig?

[tkelman]

I would have to dig into the code and/or docs for libgit2, or ask the maintainers there.

[mortenpi]

It seems that with the updated binary (the one fixing the /usr/bin thing) I am running into this even if there's no proxy. The generic binaries work fine. Output from Pkg.update etc. in a gist: https://gist.github.com/mortenpi/fcff77466e221abce2350d820bb27d9b

[staticfloat]

@mortenpi I'm unable to reproduce on a clean VM. What version of Ubuntu are you using? What is the output of dpkg -l | grep julia and dpkg -l | grep curl?

[staticfloat]

Oh, also, what is your ~/.gitconfig? That can override METADATA/.git/config.

[mortenpi]

I am running Ubuntu (Gnome) 16.04. I tried the binaries on a second machine, which should have roughly the same setup, and the binary is working fine there.

I've updated the gist. Nothing interesting in ~/.gitconfig and julia and libcurl versions seem to be the same on both machines. I also tried Pkg.initing a fresh METADATA, but it fails with the same GitError (see the gist).

[mikea]

I have installed julia from ppa in fresh ubuntu 16.10. Getting this error with any proxies:

$ julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.0 (2016-09-19 18:14 UTC)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |  x86_64-linux-gnu

julia> Pkg.add("DataFrames")
INFO: Initializing package repository /home/mike/.julia/v0.5
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
ERROR: GitError(Code:ERROR, Class:Net, Failed to resolve address for https: Name or service not known)
 in macro expansion at ./libgit2/error.jl:99 [inlined]
 in clone(::String, ::String, ::Base.LibGit2.CloneOptions) at ./libgit2/repository.jl:191
 in #clone#109(::String, ::Bool, ::Ptr{Void}, ::Nullable{Base.LibGit2.AbstractCredentials}, ::Function, ::String, ::String) at ./libgit2/libgit2.jl:327
 in (::Base.LibGit2.#kw##clone)(::Array{Any,1}, ::Base.LibGit2.#clone, ::String, ::String) at ./<missing>:0
 in (::Base.Pkg.Dir.##4#6{String,String})() at ./pkg/dir.jl:49
 in cd(::Base.Pkg.Dir.##4#6{String,String}, ::String) at ./file.jl:59
 in init(::String, ::String) at ./pkg/dir.jl:47
 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N}) at ./pkg/dir.jl:28
 in add(::String) at ./pkg/pkg.jl:100

[staticfloat]

I'm having a really hard time tracking this bug down. I took at stab at it again tonight, but I'm having difficulty figuring out where exactly the proxy configuration is getting picked up by libgit2 from within Julia. So far, in my testing, a completely from-source Julia build exhibits the same behavior as the Debian build. This is fueling my latest idea that this isn't a bug in the PPA build configuration at all, but is in fact a bug in Julia itself. At this point, I'm more surprised why the official binaries work than why the PPA binaries don't. ;)

To aid in testing, I've made some Dockerfiles that make debugging this kind of stuff much more reproducible. As an example, here's a "vanilla" build that exhibits the same behavior:

FROM ubuntu:16.10

MAINTAINER Elliot Saba

RUN apt update
RUN DEBIAN_FRONTEND=noninteractive apt install -y vim gcc g++ git make build-essential libssl-dev gfortran bar time curl m4 cmake python
RUN git clone -b release-0.5 https://github.com/JuliaLang/julia.git /root/julia
WORKDIR /root/julia
RUN make -j3

ENV PATH=$PATH:/root/julia

Saving that to a file called Dockerfile, running docker build -t julia-vanilla . and then loading up the resultant image with docker run -ti julia-vanilla /bin/bash, gives us a clean environment to start messing around with Julia. I put the following into /root/.gitconfig:

[url "https://github.com/"]
        insteadOf = git://github.com/
[https]
        proxy = http://<proxy_server>:8321
[http]
        proxy = http://<proxy_server>:8321

Where <proxy_server> is the address of a server where I'm running tinyproxy, then tried to run Pkg.init(). If I start getting requests flowing through my tinyproxy log files, then it's working, otherwise it's not. So far, the only way I've been able to get it to work is to use the official binaries, here's a Dockerfile that does that as well:

FROM ubuntu:16.10

MAINTAINER Elliot Saba

RUN apt update
RUN DEBIAN_FRONTEND=noninteractive apt install -y curl vim
RUN mkdir /root/local
WORKDIR /root/local
RUN curl -L 'https://julialang.s3.amazonaws.com/bin/linux/x64/0.5/julia-0.5.0-linux-x86_64.tar.gz' | tar --strip-components=1 -zx

WORKDIR /root
ENV PATH=$PATH:/root/local/bin

Build with docker build -t julia-official ., run with docker run -ti julia-official /bin/bash, put the .gitconfig file in again, and try to Pkg.init().