Need to test installation of packages from fresh master build

[jiahao]

From a fresh build of master and without an existing ~/.julia directory:

julia> Pkg.update()
INFO: Initializing package repository /Users/jiahao/.julia/v0.3
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl
INFO: Updating METADATA...
INFO: Computing changes...
INFO: No packages to install, update or remove

julia> Pkg.add("IJulia")
ERROR: realpath: No such file or directory
 in realpath at path.jl:127
 in mkcachedir at pkg/cache.jl:9
 in prefetch at pkg/cache.jl:34
 in resolve at pkg/entry.jl:388
 in resolve at pkg/entry.jl:363
 in edit at pkg/entry.jl:24
 in anonymous at multi.jl:1308

julia> versioninfo()
Julia Version 0.3.0-prerelease+1321
Commit bc68145* (2014-01-30 18:24 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin13.0.2)
  CPU: Intel(R) Core(TM) i5-4258U CPU @ 2.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY)
  LAPACK: libopenblas
  LIBM: libopenlibm

[stevengj]

Possibly due to #3344. @kmsquire?

[jiahao]

The function call that is causing Pkg to seize up is realpath("/Users/jiahao/.julia/v0.3/.cache"), but .cache doesn't exist anywhere. Manually creating this directory allows installation to resume but crash on a different error:

julia> Pkg.add("IJulia")
INFO: Cloning cache of BinDeps from git://github.com/JuliaLang/BinDeps.jl.git
INFO: Cloning cache of Homebrew from git://github.com/JuliaLang/Homebrew.jl.git
INFO: Cloning cache of IJulia from git://github.com/JuliaLang/IJulia.jl.git
INFO: Cloning cache of JSON from git://github.com/JuliaLang/JSON.jl.git
INFO: Cloning cache of Nettle from git://github.com/staticfloat/Nettle.jl.git
INFO: Cloning cache of REPLCompletions from git://github.com/loladiro/REPLCompletions.jl.git
INFO: Cloning cache of URIParser from git://github.com/loladiro/URIParser.jl.git
INFO: Cloning cache of ZMQ from git://github.com/JuliaLang/ZMQ.jl.git
INFO: Installing BinDeps v0.2.12
fatal: destination path 'BinDeps' already exists and is not an empty directory.
ERROR: failed process: Process(`git clone -q /Users/jiahao/.julia/v0.3/BinDeps BinDeps`, ProcessExited(128)) [128]
 in pipeline_error at process.jl:480
 in run at process.jl:457
 in install at pkg/write.jl:28
 in resolve at pkg/entry.jl:406
 in resolve at pkg/entry.jl:363
 in edit at pkg/entry.jl:24
 in anonymous at multi.jl:1308

ERROR: failed process: Process(`git clone -q /Users/jiahao/.julia/v0.3/BinDeps BinDeps`, ProcessExited(128)) [128]
 in wait at task.jl:40
 in sync_end at multi.jl:1229
 in add at pkg/entry.jl:1237
 in add at pkg/entry.jl:71
 in anonymous at pkg/dir.jl:27
 in cd at file.jl:22
 in cd at pkg/dir.jl:27
 in add at pkg.jl:20

[jiahao]

The state of the package database is also completely broken at this point:

julia> Pkg.rm("BinDeps")
INFO: No packages to install, update or remove
INFO: Package database updated

julia> Pkg.add("BinDeps")
INFO: No packages to install, update or remove
INFO: Package database updated

julia> Pkg.add("IJulia")
INFO: No packages to install, update or remove
INFO: Package database updated

julia> Pkg.rm("IJulia")
INFO: No packages to install, update or remove
INFO: Package database updated

julia> Pkg.add("IJulia")
INFO: No packages to install, update or remove
INFO: Package database updated

[jiahao]

Confirming that commit c0ccbaf (merging #3344) causes the current failure. The previous commit, 7f1c446, works fine.

[mlubin]

Could we add this test to the travis build?

[staticfloat]

I've reverted the commit for now. Adding a test for adding/removing a simple package shouldn't be too difficult, perhaps just adding/removing Example.jl or something. @kmsquire what do you think, would that catch this error?

[jiahao]

Reopening as a test issue.

[jiahao]

A test for Pkg already exists in test/pkg.jl which appears to do the equivalent of Pkg.add("Example"); Pkg.rm("Example"), so the question is why this test didn't catch the failure here.

[Ph0non]

I dont't know if this is a result of the realpath error, but there is a test, that shows, that something from the last build went wrong:

martin@aptosidbox:~/julia$ make test-pkg
    JULIA test/pkg
     * pkg
INFO: Initializing package repository /home/martin/julia/test/tmp.EMHJEONu/v0.3
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl
INFO: No packages to install, update or remove
Segmentation fault
make[1]: *** [pkg] Error 139
make: *** [test-pkg] Error 2

Citation from @simonbyrne:
"I'm seeing this as well: also make test-pkg is failing. Currently this isn't included in make testall, and so not in travis" (#5612)

[staticfloat]

Yes, it would be nice to include pkg in testall, but tests that require network access are a no-no for many distributions. If we shipped with Example.jl packaged with Julia, we could use Pkg.checkout() and point it at the local copy.

@kmsquire, @ihnorton on my local OSX machine, running 651dde7 I still see a realpath error when running make test-pkg:

$ make test-pkg
    JULIA test/pkg
     * pkg
INFO: Initializing package repository /Users/sabae/src/julia/test/tmp.EW7zSAxm/v0.3
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl
INFO: No packages to install, update or remove
ERROR: realpath: No such file or directory
 in realpath at path.jl:127
 in mkcachedir at pkg/cache.jl:18
 in prefetch at pkg/cache.jl:30
 in resolve at pkg/entry.jl:388
 in resolve at pkg/entry.jl:363
 in edit at pkg/entry.jl:24
 in anonymous at multi.jl:1308

exception on 1: ERROR: test failed: :(([keys(Pkg.installed())...]==["Example"]))
 in error at error.jl:21
 in default_handler at test.jl:19
 in do_test at test.jl:39
 in anonymous at no file:10
 in runtests at /Users/sabae/src/julia/test/testdefs.jl:5
 in anonymous at multi.jl:613
 in run_work_thunk at multi.jl:575
 in remotecall_fetch at multi.jl:647
 in remotecall_fetch at multi.jl:662
 in anonymous at multi.jl:1382
while loading pkg.jl, in expression starting on line 3
ERROR: test failed: :(([keys(Pkg.installed())...]==["Example"]))
 in error at error.jl:21
 in default_handler at test.jl:19
 in do_test at test.jl:39
 in anonymous at no file:10
 in runtests at /Users/sabae/src/julia/test/testdefs.jl:5
 in anonymous at multi.jl:613
 in run_work_thunk at multi.jl:575
 in remotecall_fetch at multi.jl:647
 in remotecall_fetch at multi.jl:662
 in anonymous at multi.jl:1382
while loading pkg.jl, in expression starting on line 3
while loading /Users/sabae/src/julia/test/runtests.jl, in expression starting on line 23

make[1]: *** [pkg] Error 1
make: *** [test-pkg] Error 2

[mlubin]

It doesn't need to be included in testall, why not just list it as an extra command in the travis script?

[staticfloat]

That's the other option I thought of, and is probably the smart thing to do first, rather than shipping with Example.jl included.

[kmsquire]

@staticfloat I'm looking now.

[kmsquire]

@staticfloat, the error in pkg-test should be fixed by 59acba9. ENV["JULIA_PKGDIR"] can't be a relative path, or it gets reinterpreted in the Pkg code.

Thanks for keeping up with this, and sorry I haven't been more responsive!

[StefanKarpinski]

JULIA_PKGDIR shouldn't need to be absolute, if it does, something is busted.

[mlubin]

Closed by http://iainnz.github.io/packages.julialang.org/?

[staticfloat]

Agreed, the core of this issue has been addressed.