Poetry ignoring credentials in source url? #6799
Comments
|
Try running Does |
neersighted
added
the
kind/question
|
@kristang I've made an MWE using a new Running |
|
Only thing I find a little odd is the url returned: I don't recall seeing that "/download/" part when I install from a private source. What service is providing your private feed? It might just be my inexperience with pypi feeds in general. |
|
I am using Azure Artifacts to store my private feed. As mentioned it works fine with |
|
Have you tried other authentication methods? I believe Azure Artifacts supports some kind of API token - last time I checked they had a guide somewhere included on the site for your feed on how to use it. |
|
@kristang That is what's being used. The tokens are used through the basic auth approach. They have no support for other authentications through pip as far as I have found, and I've been in touch with their support team regarding authentication methods. |
|
According to the documentation from Microsoft you can auth with You can then install with: pip install <package-name> --index-url https://pkgs.dev.azure.com/<your-organization-name>/<your-project-name>/_packaging/<your-feed-name>/pypi/simpleThis approach worked for me when I was testing Azure Artifacts. There is also support for using a Personal Access Token, which I assume is what you have been trying to do? |
|
A keyring would not solve the problem I've defined:
With regards to the API token versus PAT, the documentation shows that to use the |
|
Arh, sorry I completely skipped that. (Off-topic and sidenote: I would be careful about storing your credentials like that in your git) |
|
I'm facing similar problem which happened on a self-host GitLab PyPI repo too. On version |
|
I'm currently running version 1.2.1, but if this used to work in an older version I'd say it's definitely a bug that it's not working now. |
|
same problem here.. @squirrel532 with version 1.1.15 It's interesting that the authentication seems to work in the first step. Depending on which version I set in the pyproject.toml (e.g. ^0.4.0, <=0.4.0) it is able to figure out which package it should install including the correct download link. But then in the second step when it comes to downloading the package, the authentication fails. |
|
Is there any further update on this? Can it be tagged as a bug please? |
|
As a workaround, I configured https://pip.pypa.io/en/stable/topics/authentication/#netrc-support |
How do you set that up within the git repo/pyproject file? The issue I am trying to resolve revolves around the ability to have the poetry environment contain all the information needed to install the whole setup. Any settings or credentials needed outside the git/poetry files will not solve the problem. |
|
I spent the day trying to figure out how to circumvent this issue. In the end, the only thing that works is using a Instead of: [tool.poetry.dependencies]
mydep = { source = "gitlab", version = "0.1.0" }
[[tool.poetry.source]]
name = "gitlab"
url = "https://username:[email protected]/api/v4/projects/1234/packages/pypi/simple/"
secondary = trueI used: [tool.poetry.dependencies]
mydep = { git = "https://username:[email protected]/myorg/mydep.git", tag = "0.1.0" }This completely bypass the pypi server but there seems to be no other choice while this issue is not fixed. |
|
Another workaround that works for me (using a gitlab private package repository), is to put the credentials in the project With |
|
@keriksson-rosenqvist # pyproject.toml
[[tool.poetry.source]]
name = "gitlab"
url = "https://self-host-gitlab.example.com/pypi/simple/"
secondary = true# poetry.toml
[http-basic.gitlab]
username = "gitlab_user"
password = "access_token"TL;DRI traced the code and found that For someone cares about the code, here is the pseudo calling stack:
|
|
@squirrel532 Are there any special cases in the setup that could be causing this? E.g. my source name includes an underscore. Should the addition to the poetry file be done at a specific location? # pyproject.toml
[tool.poetry.dependencies]
python = "^3.9"
hello-world = { version = "^0.1.0", source = "hello_world"}
[[tool.poetry.source]]
name = "hello_world"
url = "https://[REDACTED]/pypi/simple/"
secondary = true# poetry.lock
[http-basic.hello_world]
username = "<username>"
password = "<token>"Execution output $ poetry update -vvv
Loading configuration file [REDACTED]/pypoetry/config.toml
Loading configuration file [REDACTED]/pypoetry/auth.toml
Adding repository hello_world (https://[REDACTED]/pypi/simple) and setting it as secondary
Using virtualenv: [REDACTED]/.venv
Updating dependencies
Resolving dependencies...
1: fact: poetry-credential-mwe is 0.1.0
1: derived: poetry-credential-mwe
1: fact: poetry-credential-mwe depends on hello_world (^0.1.0)
1: selecting poetry-credential-mwe (0.1.0)
1: derived: hello-world (>=0.1.0,<0.2.0)
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for [REDACTED]
[urllib3.connectionpool] Starting new HTTPS connection (1): [REDACTED]:443
[urllib3.connectionpool] https://[REDACTED]:443 "GET [REDACTED]/simple/hello-world/ HTTP/1.1" 401 343
Source (hello_world): Authorization error accessing https://[REDACTED]/pypi/simple/hello-world/
Source (hello_world): No packages found for hello-world
Source (hello_world): 0 packages found for hello-world >=0.1.0,<0.2.0
Falling back to installed packages to discover metadata for hello-world
Found 0 compatible packages for hello-world
1: fact: no versions of hello-world match >=0.1.0,<0.2.0
1: conflict: no versions of hello-world match >=0.1.0,<0.2.0
1: ! hello-world (>=0.1.0,<0.2.0) is satisfied by hello-world (>=0.1.0,<0.2.0)
1: ! which is caused by "poetry-credential-mwe depends on hello-world (^0.1.0)"
1: ! thus: version solving failed
1: Version solving took 0.400 seconds.
1: Tried 1 solutions.
.... |
|
@kristang The second file should be |
|
@squirrel532 |
|
This workaround works but I don’t think the issue should be closed because the original problem still exists. |
|
@bfontaine I think it is risky that someone places their credentials in CC @sdispater |
|
I don’t see any technical reason this shouldn’t work. Right now this is a bug: the URL including credentials is perfectly valid but Poetry appears to silently strip the credentials. It’s risky for public packages, but there are valid use-cases in a private/company environment. See what @keriksson-rosenqvist wrote in their initial message:
|
|
Hi! I also experience this issue. Thanks to @squirrel532 for this work around. So we can keep our internal projects going. I think, people should be careful with their credentials and think twice before putting them anywhere. But in our internal projects at work it is quite useful to checkout private repos which are dependent on each other. So if I understood it correctly the bug is still there but will not be fixed as the issue was closed? 🤔 |
|
Since there seems to be more people experiencing this issue, I've reopened it in hopes the bug will be looked at in the future. |
|
I cannot imagine the project will ever accept credentials in pyproject.toml, poetry has no idea where that pyproject.toml will be published, in public or private. The first project that accidentally publishes credentials in pyproject.toml will end up with a security bug against poetry. |
|
@clintonroy The question is not if the project accepts credentials because it doesn’t have a choice: URLs that contain credentials are valid HTTP URLs that shouldn’t be dealt with in a different manner than others. The suggested solution is to put the credentials in |
|
Hi!, thanks a lot for the instructions above! Here is the config command: poetry config http-basic.your_repo_alias "username" "your_token"So such steps need:
[tool.poetry.dependencies]
you_awesome_package = { source = "your_repo_alias", version = "^X.X.X" }
[[tool.poetry.source]]
name = "your_repo_alias"
url = "https://cloud-or-own-gitlab.tld/api/v4/projects/<PROJECT_ID_HERE>/packages/pypi/simple"
secondary = trueUse |
|
@fannigurt yes, but this issue is specifically about adding credentials inside the repository in order to share them with everyone. By default |
|
This issue is still present in Poetry In my testing with Google Artifact Registry (GAR), I'm seeing that the simple repository base URL is returning <!DOCTYPE html>
<html>
<head>
<title> Links for [REDACTED] </title>
</head>
<body>
<h1> Links for [REDACTED] </h1>
<a href="https://us-east1-python.pkg.dev/[REDACTED]-py3-none-any.whl#sha256=deadbeef">[REDACTED]-py3-none-any.whl</a><br/>
<a href="https://us-east1-python.pkg.dev/[REDACTED].tar.gz#sha256=feedface">[REDACTED].tar.gz</a><br/>
</body>
</html>PEP 503 states:
So there's nothing invalid about GAR's response. However, within poetry, this does not play nicely with the HTTP basic authentication credentials. When poetry processes these responses,
import urllib.parse
url='https://username:[email protected]/path/to/repo/'
href='https://us-east1-python.pkg.dev/path/to/repo/[REDACTED].tar.gz#sha256=feedface'
print(urllib.parse.urljoin(url, href))Produces: 7486 is perhaps the best solution here, as it would intercept the credentials into Poetry's authentication system early on before they have a chance to get mangled. However, there are some other potential risks caused by the same issue. For example, a user could be connecting to a repository via a proxy, and the returned absolute path would override the proxy URL. import urllib.parse
url='https://username:[email protected]/path/to/repo/'
href='https://us-east1-python.pkg.dev/path/to/repo/[REDACTED].tar.gz#sha256=feedface'
print(urllib.parse.urljoin(url, href))Produces: |
|
TLDR: check your I think I am having the same/similar issue. Unfortunately the workarounds did not work in my case. I believe in my case the error has to do with my environment (i.e. the installed packages, libraries, etc.). I'm on Ubuntu 22.04 using Poetry 1.6.1. AFAIK I haven't made any changes to poetry credential configuration between the time everything was working (~1 week ago) and now. I have traced my error to my system because (after a lot of troubleshooting), I found that running poetry commands ( Therefore I know that a) the credentials are valid and b) on a "clean" system the process works. At some point some package or library poetry uses to authenticate must have gotten corrupted in my local system however attempts to hunt this package down have all failed - downgrading poetry, force installing python and system packages like keyring, urllib, python-dbus, libdbus, etc. have all failed to solve the problem. Additionally disabling the keyring entirely and either falling back to plaintext or environment variable authentication does not work either. Unfortunately, due to the nature of the problem I cannot produce a reproduction script since I have no idea which library on my system has become "corrupted". I'm hoping someone can at least point me in a direction to look. Any help would be greatly appreciated. In the meantime I am about to run the UPDATE 2023-10-24I've been able to rule out poetry not getting the credentials from the keyring - it gets the credentials, encrypts them, and places them into the header. I haven't checked if the encryption is correct yet. The content of the response claims the token has expired but this doesn't make sense as the repository maintainers checked this and confirmed it was valid. Furthermore the issue occurs with my personal username and token too (I have developer access to the repository). Just reinstalled requests and cryptography too. Okay - I finally traced the error to conflicting credentials in SO if you come across this issue and are in a situation like mine - where none of the above workarounds fixed anything - check your |
It is 2023-10-24, not 2024 |
woow, if you create default username for token, which is |
|
I was having this same issue. Turns out I had my Gitlab credentials stored in the .netrc file as well, so @gresavage solution solved the failed authorization issues for me. |
|
I had a similar issue with GitLab. Since the package I wanted to install was in another repo in GitLab, I had to go to that repo's Settings > CI/CD and add my other repo (the one where the pipelines run) under Token Access > Limit access to this project > Groups and projects with access. More info. After that, I could use Poetry to add my private package with a pipeline job token. |
Discussed in https://github.com/orgs/python-poetry/discussions/6629
Originally posted by keriksson-rosenqvist September 26, 2022
I am trying to set up a poetry environment which pulls a certain package from a private pypi-like registry. As I am sharing the project with others via a git repository, I want the setup to be contained within the pyproject.toml file, i.e. not rely on configs, environment variables, or keyring settings that may differ between user devices.
I have therefore added a
sourcewith the basic http credentials included in the url. The exact same url is able to install the package and it's dependencies throughpip install -i https://<username:password>@[REDACTED]/pypi/simple/ my-private-packageI've set
secondary = trueas that lets dependencies be installed from the normal pypi, otherwise poetry also fails to install them.According to this issue, others have been seemingly been able to get it working with the credentials in the url, however with the setup above raises the following error for me.
Is this a bug, am I setting it up wrong, or has there been a change to the usage since the mentioned issue was able to get it to work?
The text was updated successfully, but these errors were encountered: