Poetry uses default repository instead of private repository when both repo's have same package

[janheindejong]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10
• Poetry version: 1.0.10

Issue

Hi all,

Love poetry, great stuff, etc etc.

In my company we are using a private repository for packages. We have some company specific packages there, but also builds of packages for which there are no builds on PyPI, only tarballs.

When I try to install a package that is both available on PyPI, and our private repo, I'd expect Poetry to prefer the private repo; however, this is not what seems to be happening.

My pyproject.toml:

[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.9"

[[tool.poetry.source]]
name = "private"
url = "https://private-pypi.my-company.com"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Available packages on private repository:

• pymssql-2.1.5-cp39-cp39-win32.whl
• pymssql-2.1.5-cp39-cp39-win_amd64.whl

The output of poetry add pymssql -vvv:

Using virtualenv: C:\Users\ContainerAdministrator\AppData\Local\pypoetry\Cache\virtualenvs\test5-ylwoNqwk-py3.9
PyPI: No release information found for pymssql-0.5.2, skipping
PyPI: No release information found for pymssql-0.6.0, skipping
PyPI: No release information found for pymssql-0.7.1, skipping
PyPI: No release information found for pymssql-0.7.4, skipping
PyPI: 13 packages found for pymssql *
private: 1 packages found for pymssql *
Using version ^2.1.5 for pymssql

Updating dependencies
Resolving dependencies...
   1: fact: test is 0.1.0
   1: derived: test
   1: fact: test depends on pymssql (^2.1.5)
   1: selecting test (0.1.0)
   1: derived: pymssql (^2.1.5)
PyPI: No release information found for pymssql-0.5.2, skipping
PyPI: No release information found for pymssql-0.6.0, skipping
PyPI: No release information found for pymssql-0.7.1, skipping
PyPI: No release information found for pymssql-0.7.4, skipping
PyPI: 1 packages found for pymssql >=2.1.5,<3.0.0
private: 1 packages found for pymssql >=2.1.5,<3.0.0
   1: selecting pymssql (2.1.5)
   1: Version solving took 0.269 seconds.
   1: Tried 1 solutions.

Writing lock file


Package operations: 1 install, 0 updates, 0 removals

  - Installing pymssql (2.1.5)

[EnvCommandError]
Command ['C:\\Users\\ContainerAdministrator\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\test5-ylwoNqwk-py3.9\\Scripts\\pip.exe', 'install', '--no-deps', 'pymssql==2.1.5'] errored with t
he following return code 1, and output:
Collecting pymssql==2.1.5
  Downloading pymssql-2.1.5.tar.gz (167 kB)

Lo and behold - it's downloading the tarball from PyPI, not the wheel from our private repository.

However, if I change the pyproject.toml like below, it works as expected.

[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.9"

[[tool.poetry.source]]
name = "private"
url = "https://private-pypi.my-company.com"
default = true

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Output of poetry add pymssql -vvv:

Using virtualenv: C:\Users\ContainerAdministrator\AppData\Local\pypoetry\Cache\virtualenvs\test5-ylwoNqwk-py3.9
private: 1 packages found for pymssql *
Using version ^2.1.5 for pymssql

Updating dependencies
Resolving dependencies...
   1: fact: test is 0.1.0
   1: derived: test
   1: fact: test depends on pymssql (^2.1.5)
   1: selecting test (0.1.0)
   1: derived: pymssql (^2.1.5)
private: 1 packages found for pymssql >=2.1.5,<3.0.0
   1: selecting pymssql (2.1.5)
   1: Version solving took 0.235 seconds.
   1: Tried 1 solutions.

Writing lock file


Package operations: 1 install, 0 updates, 0 removals

  - Installing pymssql (2.1.5)

Qu'est-ce que le problème? Am I missing something here?

P.s. in case you are wondering why I'm not using the latest version - that's because we're still waiting on #3490.

[areeh]

If I understand this correctly, it is also a security issue in the form of "dependency confusion" ala https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
If you upload a malicious package to PyPi that matches the name of a package in a private repo, Poetry will switch over and install the malicious package by default.

(tho since you can configure your way out of this, and the lockfile ensures you get the same package every time after you've created it, I can see how you'd consider it less of an issue)

[danmur]

I'm noticing this issue too. Annoyingly even poetry add .. --source=my-repo doesn't work either, even though it records that source in the pyproject.toml in poetry.lock it's still using the wrong repo.

[languitar]

tho since you can configure your way out of this, and the lockfile ensures you get the same package every time after you've created it, I can see how you'd consider it less of an issue

There's still the potential to end up with a malicious package on a poetry update. This is also the way the attack in the mentioned medium post was crafted. They simply injected packages with higher version numbers in the public registry and these ones would be picked up on the next update cycle.

[hankehly]

I am also facing this issue and would like to help contribute a solution. Does anyone know where the problem might be in the codebase?

[hankehly]

I solved my issue via #4523 (comment)

[abn]

@hankehly was your issue cache related? Does that mean poetry consistently picks the right package from the private repository?

[abn]

@janheindejong is this still an issue on master? The cert changes are on master too now.

[hankehly]

@hankehly was your issue cache related? Does that mean poetry consistently picks the right package from the private repository?

Yes, by deleting the local cache, poetry was able to correctly pick the right package from the private repository.

[abn]

The original reported issue should not happen with the new installer, as it chooses the package via the executor implementation.

If the issue is reproducible with poetry@master please create a new issue with relevant examples and logs.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.