fails to install from git repo when commit doesn't belong to a branch

[DetachHead]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: windows 10
• Poetry version: 1.1.12
• Link of a Gist with the contents of your pyproject.toml file: https://github.com/KotlinIsland/basedtyping/blob/master/pyproject.toml

Issue

https://github.com/KotlinIsland/basedtyping/runs/4741756977?check_suite_focus=true

  Stack trace:

  8  ~\.local\pipx\venvs\poetry\lib\site-packages\poetry\installation\executor.py:203 in _execute_operation
      201│ 
      202│             try:
    → 203│                 result = self._do_execute_operation(operation)
      204│             except EnvCommandError as e:
      205│                 if e.e.returncode == -2:

  7  ~\.local\pipx\venvs\poetry\lib\site-packages\poetry\installation\executor.py:277 in _do_execute_operation
      275│             return 0
      276│ 
    → 277│         result = getattr(self, "_execute_{}".format(method))(operation)
      278│
      279│         if result != 0:

  6  ~\.local\pipx\venvs\poetry\lib\site-packages\poetry\installation\executor.py:412 in _execute_install
      410│ 
      411│     def _execute_install(self, operation):  # type: (Install) -> None
    → 412│         return self._install(operation)
      413│
      414│     def _execute_update(self, operation):  # type: (Update) -> None

  5  ~\.local\pipx\venvs\poetry\lib\site-packages\poetry\installation\executor.py:431 in _install
      429│ 
      430│         if package.source_type == "git":
    → 431│             return self._install_git(operation)
      432│
      433│         if package.source_type == "file":

  4  ~\.local\pipx\venvs\poetry\lib\site-packages\poetry\installation\executor.py:576 in _install_git
      574│             reference = package.source_reference
      575│ 
    → 576│         git.checkout(reference, src_dir)
      577│
      578│         # Now we just need to install from the source directory

  3  ~\.local\pipx\venvs\poetry\lib\site-packages\poetry\core\vcs\git.py:281 in checkout
      279│         args += ["checkout", rev]
      280│
    → 281│         return self.run(*args)
      282│
      283│     def rev_parse(self, rev, folder=None):  # type: (str, Optional[Path]) -> str

  2  ~\.local\pipx\venvs\poetry\lib\site-packages\poetry\core\vcs\git.py:356 in run
      354│ 
      355│         return decode(
    → 356│             subprocess.check_output(
      357│                 [executable()] + list(args), stderr=subprocess.STDOUT
      358│             )

  1  ~\AppData\Local\Programs\Python\Python310\lib\subprocess.py:420 in check_output
       418│         kwargs['input'] = empty
       419│ 
    →  420│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
       421│                **kwargs).stdout
       422│

  CalledProcessError

  Command '['C:\\Program Files\\Git\\cmd\\git.exe', '--git-dir', 'C:/Users/user/basedtyping/.venv/src/basedmypy/.git', '--work-tree', 'C:/Users/user/basedtyping/.venv/src/basedmypy', 'checkout', 'e5d40fc37d8d6eca75d8784020d82b4574b042b7']' returned non-zero exit status 128.

  at ~\.local\pipx\venvs\poetry\lib\site-packages\poetry\utils\_compat.py:217 in run
      213│                 process.wait()
      214│                 raise
      215│             retcode = process.poll()
      216│             if check and retcode:
    → 217│                 raise CalledProcessError(
      218│                     retcode, process.args, output=stdout, stderr=stderr
      219│                 )
      220│         finally:
      221│             # None because our context manager __exit__ does not use them.

[synrg]

This looks like more of the same, using poetry 1.2.0a2 here.

Gist of the pyproject.toml: https://gist.github.com/synrg/3fb23744614b50da8f080f5c566846df

Note that the Red-DiscordBot dependency, in turn, depends on a PR for Red-Lavalink. The git command poetry tries to execute fails, as it can't checkout this pull which has not yet been fetched:

  CalledProcessError

  Command '['git', '--git-dir', '/tmp/pypoetry-git-Red-Lavalinkwr2dmj40/.git', '--work-tree', '/tmp/pypoetry-git-Red-Lavalinkwr2dmj40', 'checkout', 'refs/pull/99/merge']' returned non-zero exit status 1.

  at /usr/local/lib/python3.9/subprocess.py:528 in run
       524│             # We don't call process.wait() as .__exit__ does that for us.
       525│             raise
       526│         retcode = process.poll()
       527│         if check and retcode:
    →  528│             raise CalledProcessError(retcode, process.args,
       529│                                      output=stdout, stderr=stderr)
       530│     return CompletedProcess(process.args, retcode, stdout, stderr)
       531│
       532│

i.e. git checkout refs/pull/99/merge won't work without first being fetched.

[abn]

@DetachHead @synrg can you verify if #5428 fixes this issue?

[synrg]

@DetachHead @synrg can you verify if #5428 fixes this issue?

Unfortunately I can't easily do that, due to the following circumstances:

• Red-DiscordBot development has progressed, and now V3/dpy2_unstable has been deleted and all of the work in it merged into V3/develop, so my testing is not on the exact same thing as before.
• I can't find where V3/dpy2_unstable used to point. i think the devs did something on a local branch, obscuring this.
• Even with the current 1.2.0b1, if I switch my pyproject.toml to point at V3/develop, the install succeeds (i.e. apparently it now depends on an rc release of Red-LavaLink instead of a PR as before)
• I also tried the use-dulwich branch of poetry and it succeeds too; but this tells us nothing, since the original test case is now destroyed

Sorry about that.

[synrg]

@abn Good news / bad news.

With the help of a Red dev who recovered an older version of the branch: https://github.com/Kowlin/Red-DiscordBot/tree/V3/dpy2_unstable I was able to reproduce the issue using 1.2.0b1.

Unfortunately, when using poetry from your use-dulwich branch, it breaks again, only this time with a slightly different error, an AssertionError Invalid object name b'dpy2_unstable\n'

[synrg]

Here's the new gist for the pyproject.toml that caused the failure shown in the previous comment. https://gist.github.com/synrg/5f8bbe51b29315abcbbb6ae36de5c693

[abn]

Thanks @synrg this helps. I'll do some checking.

[abn]

@synrg Looks like the issue you encountered was caused by the dependency to Red-Lavalink as the branch nop longer exists. I have improved the error handling when that happens on my branch.

$ poetry add git+https://github.com/Rapptz/discord.py#eca472759338d87ad616a44191b7a2ccf55bbabf
Creating virtualenv foobar in /tmp/foobar/.venv

Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 10 installs, 0 updates, 0 removals

  • Installing frozenlist (1.3.0)
  • Installing idna (3.3)
  • Installing multidict (6.0.2)
  • Installing aiosignal (1.2.0)
  • Installing async-timeout (4.0.2)
  • Installing attrs (21.4.0)
  • Installing charset-normalizer (2.0.12)
  • Installing yarl (1.7.2)
  • Installing aiohttp (3.8.1)
  • Installing discord.py (2.0.0a3952+geca47275 eca4727)
$ poetry add git+https://github.com/Kowlin/Red-DiscordBot#V3/dpy2_unstable 

Updating dependencies
Resolving dependencies... (0.6s)

Failed to clone 'dpy2_unstable' from https://github.com/Cog-Creators/Red-Lavalink

[synrg]

Aha. That makes sense. So a more thorough test would've had to point at a PR that still exists, and that is expected to work? I wasn't aware that both the branch and the PR were now gone. If contriving such a second test would help, let me know and I'll see what can be done.

[synrg]

I have updated our test to use this pyproject.toml which is a more complete restore of the Red-DiscordBot dpy2_unstable branch and the Red-Lavalink dpy2_unstable branch, and the PR on that branch (both the branch and the PR exist now):

https://gist.github.com/synrg/61642721c91141b1377c056b8ddf100f

Then I tried both on 1.2.0b1 (fails as it did before) and your updated use-dulwich branch, which I deleted locally and checked out again. Now it fails with this error:

synrg@bgpc:~/work/dronefly$ poetry lock
Updating dependencies
Resolving dependencies... (104.5s)

Failed to clone 'refs/pull/2/merge' from https://github.com/jack1142/Red-Lavalink

Note that it's no longer failing on the branch, because it now exists. But it's not successfully cloning the PR, even though it exists now.

[abn]

So, turns out this is because poetry determins refs/pull/2/merge to be a revision not a branch/ref. Added some awareness to my branch to handle that too. And well, a test case to boot.

https://github.com/python-poetry/poetry/pull/5428/files#diff-9fc14762c1e5b7ebd6d4e53faf2bc33739657e108770f580ecaa69b23d9fe82fR163-R167

With the fix, seems things are happy.

$ poetry add git+https://github.com/jack1142/Red-DiscordBot#branch_for_sb 

Updating dependencies
Resolving dependencies... (47.3s)

Writing lock file

Package operations: 37 installs, 0 updates, 0 removals

  • Installing idna (3.2)
  • Installing multidict (5.1.0)
  • Installing async-timeout (3.0.1)
  • Installing attrs (21.2.0)
  • Installing chardet (4.0.0)
  • Installing typing-extensions (3.10.0.2)
  • Installing yarl (1.6.3)
  • Installing aiohttp (3.7.4.post0)
  • Installing pycparser (2.20)
  • Installing cffi (1.14.6)
  • Installing colorama (0.4.4)
  • Installing commonmark (0.9.1)
  • Installing contextlib2 (21.6.0)
  • Installing discord.py (2.0.0a3952+geca47275 3d914e0)
  • Installing pygments (2.10.0)
  • Installing pytz (2021.1)
  • Installing red-commons (1.0.0)
  • Installing six (1.16.0)
  • Installing aiohttp-json-rpc (0.13.3)
  • Installing aiosqlite (0.17.0)
  • Installing appdirs (1.4.4)
  • Installing apsw-wheels (3.36.0.post1)
  • Installing babel (2.9.1)
  • Installing click (8.0.1)
  • Installing distro (1.6.0)
  • Installing fuzzywuzzy (0.18.0)
  • Installing markdown (3.3.4)
  • Installing psutil (5.8.0)
  • Installing pynacl (1.4.0)
  • Installing python-dateutil (2.8.2)
  • Installing python-levenshtein-wheels (0.13.2)
  • Installing pyyaml (5.4.1)
  • Installing red-lavalink (0.11.0rc0 94526e2)
  • Installing rich (10.9.0)
  • Installing schema (0.7.4)
  • Installing uvloop (0.16.0)
  • Installing red-discordbot (3.5.0.dev1 1e2a710)

[DetachHead]

still getting the same error with #5428

(bitsus-XrR4RrAa-py3.10) PS C:\Users\amogus\bitsus> pipx install --suffix=@5428 'poetry @ git+https://github.com/python-poetry/poetry.git@refs/pull/5428/head'
  installed package poetry 1.2.0b1 (poetry@5428), installed using Python 3.10.1
  These apps are now globally available
    - [email protected]
done! ✨ 🌟 ✨
(bitsus-XrR4RrAa-py3.10) PS C:\Users\amogus\bitsus> poetry update
Updating dependencies
Resolving dependencies...

  CalledProcessError

  Command '['C:\\Program Files\\Git\\cmd\\git.exe', '--git-dir', 'C:/Users/amogus/AppData/Local/Temp/pypoetry-git-basedtypin45669d59/.git', '--work-tree', 'C:/Users/amogus/AppData/Local/Temp/pypoetry-git-basedtypinudj4qdir', 'checkout', '7962a3380a906d3c1e17bec13461cb41e7ee2744']' returned non-zero exit status 128.

  at ~\.local\pipx\venvs\poetry\lib\site-packages\poetry\utils\_compat.py:217 in run
      213│                 process.wait()
      214│                 raise
      215│             retcode = process.poll()
      216│             if check and retcode:
    → 217│                 raise CalledProcessError(
      218│                     retcode, process.args, output=stdout, stderr=stderr
      219│                 )
      220│         finally:
      221│             # None because our context manager __exit__ does not use them.

[abn]

@DetachHead looks like you installed the PR version but used your default poetry.

[email protected] update

[DetachHead]

my bad, i didnt realize that's what --suffix does

now i get this error

> poetry@5428 update
Updating dependencies
Resolving dependencies...

  KeyError

  b'7962a3380a906d3c1e17bec13461cb41e7ee2744'

  at ~\.local\pipx\venvs\poetry@5428\lib\site-packages\dulwich\object_store.py:584 in get_raw
       580│             try:
       581│                 return alternate.get_raw(hexsha)     
       582│             except KeyError:
       583│                 pass
    →  584│         raise KeyError(hexsha)
       585│
       586│     def add_objects(self, objects, progress=None):   
       587│         """Add a set of objects to this object store.

my pyproject.toml:

[tool.poetry.dependencies]
basedtyping = { git = "https://github.com/KotlinIsland/basedtyping.git", rev = "7962a3380a906d3c1e17bec13461cb41e7ee2744" }

[synrg]

excellent. i can confirm with those commits, my test case no longer fails, and all of the dependencies are successfully installed, including the PR for Red-Lavalink. thanks!

[abn]

@DetachHead I suspect that might be out of scope for us in general. As what you are seeing is a side-effect of github not running git gc on the server side. This means you can see the commit via ui. I am not entirely sure how to fetch the refspecs locally for these cases, atleast not in a consistent manner.

$ git clone https://github.com/KotlinIsland/basedtyping.git
Cloning into 'basedtyping'...
remote: Enumerating objects: 521, done.
remote: Counting objects: 100% (521/521), done.
remote: Compressing objects: 100% (315/315), done.
remote: Total 521 (delta 305), reused 400 (delta 193), pack-reused 0
Receiving objects: 100% (521/521), 102.83 KiB | 2.71 MiB/s, done.
Resolving deltas: 100% (305/305), done.
$ cd basedtyping/
$ git checkout 7962a3380a906d3c1e17bec13461cb41e7ee2744
fatal: reference is not a tree: 7962a3380a906d3c1e17bec13461cb41e7ee2744
$ git reflog
0b07cdf (HEAD -> master, origin/master, origin/HEAD) HEAD@{0}: clone: from https://github.com/KotlinIsland/basedtyping.git

[DetachHead]

i know npm is able to install commits that don't belong to a branch, however i did notice the commit from my original post is gone so i suppose they do run git gc occasionally. in which case it's a terrible idea to pin dependencies to such commits so it's fair enough to not want to support that

i'm curious though as to how my scenatio is different to @abn and @synrg's?

[abn]

@DetachHead maybe a post merge branch that was not deleted? Or another branch existed off of it? I suspect there are a variety of possibilitie there unfortunately.

[abn]

Resolved-by: #5428

[LVladymyr]

The problem still reproducible (for poetry 1.1.14) :
Command '['C:\\Program Files\\Git\\cmd\\git.exe', '--git-dir', 'C:/.../AppData/Local/Temp/pypoetry-git-user-permission-clien1x0ug_35/.git', '--work-tree', 'C:/.../AppData/Local/Temp/pypoetry-git-user-permission-clien1x0ug_35', 'checkout', 'master']' returned non-zero exit status 1.
This command fail with message

fatal: not a git repository: 'C:/.../AppData/Local/Temp/pypoetry-git-user-permission-clien4ahexqpt/.git'

As workaround I use HEAD to avoid useless checkout here:
poetry add git+ssh://[email protected]:account/repository-name.git#HEAD

[Jackenmen]

@LVladymyr The linked PR is in 1.2 milestone so there was no stable release of poetry with the fix yet. It should not be reproducible in latest 1.2.0 beta release.

[m-danya]

I've faced the same issue with the following line:

fasttext = {git = "https://github.com/cfculhane/fastText"}

It worked with poetry==1.3.2, but doesn't work with poetry=1.11.1, since.... drum roll.... the branch is named "main" and the old poetry by default use "master" branch. So it started working after I specified the commit:

fasttext = {git = "https://github.com/cfculhane/fastText", rev="4a44513"}

I think error messages can be greatly improved to show the reason of a failure like "Couldn't checkout branch 'master'".

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.