Custom repository doesn't have precedence over PyPI

[jaklan]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS Mojave 10.14.6
• Poetry version: 1.1.3
• Link of a Gist with the contents of your pyproject.toml file:

[[tool.poetry.source]]
name = 'foobar'
url = 'http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple'

[build-system]
requires = ["poetry_core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Issue

In docs we can read:

Any custom repository will have precedence over PyPI.

In the above example, foobar is a proxy repo incl. both internal packages and the public ones from PyPI. Running poetry install should cause a creation of poetry.lock, where all the packages have foobar in their source field. However, it's completely opposite - none of the packages has a source field, which means all of them are downloaded directly from PyPI.

When I add default = true to the above source element, which means PyPI is disabled, everything works properly - generated poetry.lock includes foobar as a source for all packages and installation doesn't break, so it's not a problem with repository itself.

It's especially frustrating when you want to use poetry export (for any reason), because then the final requirements.txt doesn't include neither --index-url nor --extra-index-url (it seems it depends on the actually used sources, not the pyproject.toml content).

[jaklan]

I've just found the very similar issues:
#1677
#2564
but the problem seems not to be resolved for almost a year, so I will keep this issue open to bump the topic.

[abn]

Can you please try the fix at #3251.

Using pipx

pipx install --suffix=@3251 'poetry @ git+https://github.com/python-poetry/poetry.git@refs/pull/3251/head'

Using a container (podman | docker)

podman run --rm -i --entrypoint bash python:3.8 <<EOF
set -xe
python -m pip install -q git+https://github.com/python-poetry/poetry.git@refs/pull/3251/head
install -d foobar
pushd foobar
cat > pyproject.toml <<TOML
[tool.poetry]
name = "foobar"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[[tool.poetry.source]]
name = 'testpypi'
url = 'https://test.pypi.org/simple'

[tool.poetry.dependencies]
python = "^3.8"
py-serializer = "0.9.5"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
TOML
poetry lock
poetry export -f requirements.txt
EOF

[benjaminlee314]

is this the same issue as #3249 ?

[jaklan]

@abn I've jus tested the PR. poetry.lock seems good, there's a proper [package.source] table for all packages:

[package.source]
type = "legacy"
url = "http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple"
reference = 'foobar'

But... there's one more thing. I've also run:
poetry@3251 export -f requirements.txt -o requirements.txt --without-hashes --with-credentials
and the first line of the output file is:
--extra-index-url http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple

When using default = true we get then:
--index-url http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple

At first sight it seems okay, but we have to remember about the issue with pip and --extra-index-url:
https://pydist.com/blog/extra-index-url
pypa/pip#5045
I assume it's already handled by Poetry itself, but to make it error-prone when exporting the requirements.txt file imho we should also keep the approach of specifying individual index per each package, e.g.:
--index-url http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple anyconfig==0.9.11

PS Of course there should be also --trusted-host flag in any of these scenarios due to http, but there was another PR for that, afair - already merged, just not released yet.

[abn]

@jaklan appreciate the input on that. I'd suggest we raise that as a new issue. I agree that a per package index is better for the extra-index case. One worry, however, is that people seem to be attached to the current export format because some do text processing of the output it for various reasons.

[jaklan]

@abn here it is: #3269.

[abn]

Resolved with 1.1.4.

[jaklan]

There is a regression in 1.1.4 caused by the above fix:
#3306

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.