Problems authenticating to a private index

[fgsalomon]

Pre-submission checks

• I am not filing an auditing error (false positive or negative). These must be reported to pypa/advisory-database instead.
• I agree to follow the PSF Code of Conduct.
• I have looked through the open issues for a duplicate report.

Expected behavior

Hi,

I have a project with a dependency on a package hosted in a private index. The private index is a Google Artifact Registry.
This project uses a requirements.txt file to handle the dependencies.

I'm authenticating through the keyring with the Google Artifact Registry backend. I'm authenticated and have the right permissions in Google Cloud.

I can install my private package without issue by providing the extra index url:

pip install --extra-index-url MY_INDEX_URL -r requirements/requirements.txt 

However, when I run pip-audit with --extra-index-url it can't find the package:

pip-audit -vvvv --extra-index-url MY_INDEX_URL -r requirements/requirements.txt

I expected pip-audit to be able to analyze the dependencies (at least the public ones)

Actual behavior

pip-audit returns an error because it could not find the private package

Reproduction steps

1. Have a requirements.txt file with a package hosted in a Google Artifact Registry with a correct setup of the keyring
2. Run pip-audit -vvvv --extra-index-url MY_INDEX_URL -r requirements/requirements.txt

Logs

DEBUG:pip_audit._cli:parsed arguments: Namespace(local=False, requirements=[<_io.TextIOWrapper name='requirements/requirements.txt' mode='r' encoding='UTF-8'>], project_path=None, format=<OutputFormatChoice.Columns: 'columns'>, vulnerability_service=<VulnerabilityServiceChoice.Pypi: 'pypi'>, dry_run=False, strict=False, desc=<VulnerabilityDescriptionChoice.Auto: 'auto'>, aliases=<VulnerabilityAliasChoice.Auto: 'auto'>, cache_dir=None, progress_spinner=<ProgressSpinnerChoice.On: 'on'>, timeout=15, paths=[], verbose=4, fix=False, require_hashes=False, index_url=None, extra_index_urls=['MY_INDEX_URL'], skip_editable=False, no_deps=False, output=PosixPath('stdout'), ignore_vulns=[], disable_pip=False)
ERROR:pip_audit._virtual_env:internal pip failure:  [...]
ERROR: Could not find a version that satisfies the requirement MY_PRIVATE_PACKAGE==X.Y.Z (from versions: none)
ERROR: No matching distribution found for MY_PRIVATE_PACKAGE==X.Y.Z

ERROR:pip_audit._cli:Failed to install packages: ['/var/folders/nl/jq_nzg654wn573pkhr9949xh0000gn/T/tmpful3a4s9/bin/python3.11', '-m', 'pip', 'install', '--no-input', '--extra-index-url', 'MY_INDEX_URL', '--dry-run', '--report', '/var/folders/nl/jq_nzg654wn573pkhr9949xh0000gn/T/tmpn0nqqkdw/tmpcz4kjwr9', '-r', 'requirements/requirements.txt']

Additional context

No response

OS name, version, and architecture

Mac OS 14.2.1 Apple Silicon & Ubuntu 22.04 x86_64

pip-audit version

2.7.1

pip version

24.0

Python version

3.11

[woodruffw]

Thanks for the report @fgsalomon! And thanks for filling out the full template, I greatly appreciate it.

I'm looking into this now -- pip-audit should transparently delegate things like keyring authentication to pip, so this is potentially an error in how we do that.

[woodruffw]

Current operating theory: we added --no-input as a fix for #706, which appears to have disabled keyring lookups by default as of pip 23.1 (pypa/pip#11698).

I think what we need to do here is pass --keyring-provider=subprocess or similar, with the presumption that the user has keyring installed somewhere on their $PATH. We can't use --keyring-provider=import by default most likely, since our default requirements behavior is to create a clean virtual environment (which won't have the keyring package in it).

@fgsalomon can you confirm that keyring is on your $PATH? If so, I think that's the most viable path forwards here, and I'll make a PR for you to test against 🙂

[woodruffw]

Additional pip context: pypa/pip#8719

[fgsalomon]

Yes, keyring is on my $PATH.

[woodruffw]

Thanks @fgsalomon! I've opened #743 as a prospective fix -- could you give the changes on that branch a try and see if they resolve the issue for you? If so, I'll get them merged and do a patch release ASAP.

[fgsalomon]

It didn't work, I got the same error.
FYI, I'm using a virtual env and keyring is installed in it (my $PATH includes the virtual env). Could be this a problem?

I've put a print to see the pip command output:

[...]
Looking in indexes: https://pypi.org/simple, MY_INDEX_URL/
2 location(s) to search for versions of MY_PRIVATE_PACKAGE:
* https://pypi.org/simple/MY_PRIVATE_PACKAGE/
* MY_INDEX_URL/MY_PRIVATE_PACKAGE/
Fetching project page and analyzing links: https://pypi.org/simple/MY_PRIVATE_PACKAGE/
Getting page https://pypi.org/simple/MY_PRIVATE_PACKAGE/
Found index url https://pypi.org/simple/
Looking up "https://pypi.org/simple/MY_PRIVATE_PACKAGE/" in the cache
Request header has "max_age" as 0, cache bypassed
No cache entry available
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/MY_PRIVATE_PACKAGE/ HTTP/1.1" 404 13
Status code 404 not in (200, 203, 300, 301, 308)
Could not fetch URL https://pypi.org/simple/MY_PRIVATE_PACKAGE/: 404 Client Error: Not Found for url: https://pypi.org/simple/MY_PRIVATE_PACKAGE/ - skipping
Fetching project page and analyzing links: MY_INDEX_URL/MY_PRIVATE_PACKAGE/
Getting page MY_INDEX_URL/MY_PRIVATE_PACKAGE/
Found index url MY_INDEX_URL/
Looking up "MY_INDEX_URL/MY_PRIVATE_PACKAGE/" in the cache
Request header has "max_age" as 0, cache bypassed
No cache entry available
Starting new HTTPS connection (1): MY_INDEX_URL:443
MY_INDEX_URL:443 "GET /simple/MY_PRIVATE_PACKAGE/ HTTP/1.1" 401 60
Found index url MY_INDEX_URL/
Keyring provider requested: subprocess
Keyring provider set: subprocess with executable /Users/fgsalomon/myproject/venv/bin/MY_INDEX_URL
Status code 401 not in (200, 203, 300, 301, 308)
Could not fetch URL MY_INDEX_URL/MY_PRIVATE_PACKAGE/: 401 Client Error: Unauthorized for url: 
MY_INDEX_URL/MY_PRIVATE_PACKAGE/ - skipping
Skipping link: not a file: https://pypi.org/simple/MY_PRIVATE_PACKAGE/
Skipping link: not a file: MY_INDEX_URL/MY_PRIVATE_PACKAGE/
Given no hashes to check 0 links for project 'MY_PRIVATE_PACKAGE': discarding no candidates