[poetry install] Error resolving packages from virtual repositories

[tgarciaalv]

Description

Hello. We have a internal pip mirror to download python packages, configured as a JFrog virtual repository. Since we updated to 1.8.0 or 1.8.1 we got a failure resolving the packages when executing poetry install command.
We get ` Package('xxx-lib', '0.17.0') is not in list `  when actually it is. 

Setting the real repository URL (instead of the virtual repo url) works, but we need to use the virtual repo url.

Workarounds

fix the version to Poetry (version 1.7.1)

Poetry Installation Method

pipx

Operating System

ubuntu

Poetry Version

Poetry (version 1.8.1)

Poetry Configuration

cache-dir = "/root/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
repositories.matillion.url = "https://<my jfrog virtual repo>/simple"
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /root/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

No response

Example pyproject.toml

No response

Poetry Runtime Logs

pip install poetry --progress-bar off
poetry config virtualenvs.create true
poetry config virtualenvs.in-project true
poetry config http-basic.<my-organisation> ${JFROG_USER} ${JFROG_TOKEN}
poetry config --list
poetry -vvv install --no-ansi --no-interaction --no-root

[...]

[urllib3:urllib3.connectionpool] https://xxxxxxx.jfrog.io:443 "GET /artifactory/api/xxxxxxx/xxx-lib/0.17.0/xxx_lib-0.17.0-py3-none-any.whl HTTP/1.1" 206 10001
   1: Version solving took 14.901 seconds.
   1: Tried 1 solutions.

  ValueError

  Package('xxx-lib', '0.17.0') is not in list

  at /usr/local/lib/python3.11/site-packages/poetry/repositories/legacy_repository.py:66 in package
       62│         Note that this will be cached so the subsequent operations
       63│         should be much faster.
       64│         """
       65│         try:
    →  66│             index = self._packages.index(Package(name, version))
       67│ 
       68│             return self._packages[index]
       69│         except ValueError:
       70│             package = super().package(name, version, extras)

The following error occurred when trying to handle this error:


  Stack trace:

  29  /usr/local/lib/python3.11/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  28  /usr/local/lib/python3.11/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 

  27  /usr/local/lib/python3.11/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 

  26  /usr/local/lib/python3.11/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code

  25  /usr/local/lib/python3.11/site-packages/cleo/application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  24  /usr/local/lib/python3.11/site-packages/cleo/commands/base_command.py:117 in run
       115│         io.input.validate()
       116│ 
     → 117│         return self.execute(io) or 0
       118│ 
       119│     def merge_application_definition(self, merge_args: bool = True) -> None:

  23  /usr/local/lib/python3.11/site-packages/cleo/commands/command.py:61 in execute
        59│ 
        60│         try:
     →  61│             return self.handle()
        62│         except KeyboardInterrupt:
        63│             return 1

  22  /usr/local/lib/python3.11/site-packages/poetry/console/commands/install.py:153 in handle
       151│         self.installer.verbose(self.io.is_verbose())
       152│ 
     → 153│         return_code = self.installer.run()
       154│ 
       155│         if return_code != 0:

  21  /usr/local/lib/python3.11/site-packages/poetry/installation/installer.py:104 in run
       102│             self.verbose(True)
       103│ 
     → 104│         return self._do_install()
       105│ 
       106│     def dry_run(self, dry_run: bool = True) -> Installer:

  20  /usr/local/lib/python3.11/site-packages/poetry/installation/installer.py:241 in _do_install
       239│                 source_root=self._env.path.joinpath("src")
       240│             ):
     → 241│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       242│         else:
       243│             self._io.write_line("Installing dependencies from lock file")

  19  /usr/local/lib/python3.11/site-packages/poetry/puzzle/solver.py:71 in solve
        69│         with self._progress(), self._provider.use_latest_for(use_latest or []):
        70│             start = time.time()
     →  71│             packages, depths = self._solve()
        72│             end = time.time()
        73│ 

  18  /usr/local/lib/python3.11/site-packages/poetry/puzzle/solver.py:154 in _solve
       152│ 
       153│         try:
     → 154│             result = resolve_version(self._package, self._provider)
       155│ 
       156│             packages = result.packages

  17  /usr/local/lib/python3.11/site-packages/poetry/mixology/__init__.py:18 in resolve_version
        16│     solver = VersionSolver(root, provider)
        17│ 
     →  18│     return solver.solve()
        19│ 

  16  /usr/local/lib/python3.11/site-packages/poetry/mixology/version_solver.py:175 in solve
       173│             while next is not None:
       174│                 self._propagate(next)
     → 175│                 next = self._choose_package_version()
       176│ 
       177│             return self._result()

  15  /usr/local/lib/python3.11/site-packages/poetry/mixology/version_solver.py:514 in _choose_package_version
       512│             package = locked
       513│ 
     → 514│         package = self._provider.complete_package(package)
       515│ 
       516│         conflict = False

  14  /usr/local/lib/python3.11/site-packages/poetry/puzzle/provider.py:489 in complete_package
       487│                 dependency_package = DependencyPackage(
       488│                     dependency,
     → 489│                     self._pool.package(
       490│                         package.pretty_name,
       491│                         package.version,

  13  /usr/local/lib/python3.11/site-packages/poetry/repositories/repository_pool.py:204 in package
       202│         for repo in self.repositories:
       203│             try:
     → 204│                 return repo.package(name, version, extras=extras)
       205│             except PackageNotFound:
       206│                 continue

  12  /usr/local/lib/python3.11/site-packages/poetry/repositories/legacy_repository.py:70 in package
        68│             return self._packages[index]
        69│         except ValueError:
     →  70│             package = super().package(name, version, extras)
        71│             package._source_type = "legacy"
        72│             package._source_url = self._url

  11  /usr/local/lib/python3.11/site-packages/poetry/repositories/cached_repository.py:75 in package
        73│         extras: list[str] | None = None,
        74│     ) -> Package:
     →  75│         return self.get_release_info(canonicalize_name(name), version).to_package(
        76│             name=name, extras=extras
        77│         )

  10  /usr/local/lib/python3.11/site-packages/poetry/repositories/cached_repository.py:52 in get_release_info
        50│             return PackageInfo.load(self._get_release_info(name, version))
        51│ 
     →  52│         cached = self._release_cache.remember(
        53│             f"{name}:{version}", lambda: self._get_release_info(name, version)
        54│         )

   9  /usr/local/lib/python3.11/site-packages/poetry/utils/cache.py:147 in remember
       145│         value = self.get(key)
       146│         if value is None:
     → 147│             value = callback() if callable(callback) else callback
       148│             self.put(key, value, minutes)
       149│         return value

   8  /usr/local/lib/python3.11/site-packages/poetry/repositories/cached_repository.py:53 in <lambda>
        51│ 
        52│         cached = self._release_cache.remember(
     →  53│             f"{name}:{version}", lambda: self._get_release_info(name, version)
        54│         )
        55│ 

   7  /usr/local/lib/python3.11/site-packages/poetry/repositories/legacy_repository.py:123 in _get_release_info
       121│         yanked = page.yanked(name, version)
       122│ 
     → 123│         return self._links_to_data(
       124│             links,
       125│             PackageInfo(

   6  /usr/local/lib/python3.11/site-packages/poetry/repositories/http_repository.py:360 in _links_to_data
       358│ 
       359│         # drop yanked files unless the entire release is yanked
     → 360│         info = self._get_info_from_links(links, ignore_yanked=not data.yanked)
       361│ 
       362│         data.summary = info.summary

   5  /usr/local/lib/python3.11/site-packages/poetry/repositories/http_repository.py:310 in _get_info_from_links
       308│                 return self._get_info_from_metadata(
       309│                     universal_python3_wheel
     → 310│                 ) or self._get_info_from_wheel(universal_python3_wheel)
       311│ 
       312│             if universal_python2_wheel:

   4  /usr/local/lib/python3.11/site-packages/poetry/repositories/http_repository.py:123 in _get_info_from_wheel
       121│             try:
       122│                 package_info = PackageInfo.from_metadata(
     → 123│                     metadata_from_wheel_url(link.filename, link.url, self.session)
       124│                 )
       125│             except HTTPRangeRequestUnsupported:

   3  /usr/local/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:77 in metadata_from_wheel_url
        75│         # After context manager exit, wheel.name will point to a deleted file path.
        76│         # Add `delete_backing_file=False` to disable this for debugging.
     →  77│         with LazyWheelOverHTTP(url, session) as lazy_file:
        78│             metadata_bytes = lazy_file.read_metadata(name)
        79│ 

   2  /usr/local/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:289 in __enter__
       287│     def __enter__(self: U) -> U:
       288│         super().__enter__()
     → 289│         self._setup_content()
       290│         return self
       291│ 

   1  /usr/local/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:357 in _setup_content
       355│         if self._length is None:
       356│             logger.debug("begin fetching content length")
     → 357│             self._length = self._fetch_content_length()
       358│             logger.debug("done fetching content length (is: %d)", self._length)
       359│             # Enable us to seek and write anywhere in the backing file up to this

  AssertionError

  

  at /usr/local/lib/python3.11/site-packages/poetry/inspection/lazy_wheel.py:526 in _fetch_content_length
      522│             # If we *could* download some file contents, then write them to the end of
      523│             # the file and set up our bisect boundaries by hand.
      524│             with self._stay():
      525│                 response_length = int(tail.headers["Content-Length"])
    → 526│                 assert response_length == min(initial_chunk_size, ret_length)
      527│                 self.seek(-response_length, io.SEEK_END)
      528│                 # Default initial chunk size is currently 1MB, but streaming content
      529│                 # here allows it to be set arbitrarily large.
      530│                 for chunk in tail.iter_content(CONTENT_CHUNK_SIZE):

Exited with code exit status 1

[radoering]

Edit: If you are reading this later, don't use this command, use #9056 (comment)

Can you provide the output of the following command (replacing <link-to-wheel>)?

curl -sv -H "Range: bytes=-10000" --stderr - -o /dev/null -w 'Response-Length: %{size_download}\n' <link-to-wheel> | grep 'Length\|Range'

That should at least show what exactly is wrong with the server response.

[tgarciaalv]

Can you provide the output of the following command (replacing )?

curl -sv -H "Range: bytes=-10000" --stderr - -o /dev/null -w 'Response-Length: %{size_download}\n' <link-to-wheel> | grep 'Length\|Range'

That should at least show what exactly is wrong with the server response.

Hello. This is the output:

> Range: bytes=-10000
< Content-Length: 91
Response-Length: 91

[radoering]

Unfortunately, the curl command apparently was not good enough. I suppose you got a negative response (length of data is too short) and some information might be missing.

Can you try the following updated command?

curl -sv -H "Range: bytes=-10000" --stderr - -o /dev/null -w 'Response-Length: %{size_download}\nHTTP-Code:%{http_code}\n' <link-to-file> | grep -i 'Length\|Range\|Code'

and if the HTTP-Code is 401 or similar provide authentication to the command, e.g. via -u <user>:<password>.

[tgarciaalv]

Sorry, you are right. Here it is a 20x response:

`➜ ~ curl -u user:passwd -sv -H "Range: bytes=-10000" --stderr - -o /dev/null -w 'Response-Length: %{size_download}\nHTTP-Code:%{http_code}\n' https://xxxxxxxxxxxxx-any.whl | grep -i 'Length|Range|Code'

Range: bytes=-10000
< Content-Length: 10001
< Accept-Ranges: bytes
< Content-Range: bytes -4749-5251/5252
Response-Length: 5252
HTTP-Code:206`

[radoering]

That helps a lot. With this information we can handle Artifactory's nonconform behavior better.

Nevertheless, if anyone eligable for Artifactory support is reading this, it may still make sense to report the following bugs (because poetry lock will be faster if they were fixed):

It seems JFrog tried to fix range requests with negative offsets in RTDEV-38572 (see changelog):

Fixed an issue whereby, when using a curl command with a ‘range’ HTTP header to fetch bytes from the end of a text file, Artifactory fetched bytes from the beginning of the file instead.

However, there are still two issues:

Issue 1 (requested bytes < total size)

Artifactory still returns the wrong number of bytes. It returns one more byte than requested:

$ curl -sv -H "Range: bytes=-1000" --stderr - -o /dev/null -w 'Response-Length: %{size_download}\nHTTP-Code:%{http_code}\n' <url-redacted> | grep -i 'Length\|Range\|Code'
> range: bytes=-1000
< content-length: 1001
< accept-ranges: bytes
< content-range: bytes 16296-17296/17297
Response-Length: 1001
HTTP-Code:206

It should return 1000 bytes instead of 1001.
content-length should be 1000.
content-range should be 16297-17296/17297

Issue 2 (requested bytes > total size)

That's what we do not handle robust enough in Poetry right now. If we request more bytes than the size of the artifact, Artifactory returns a wrong content-length (and a strange content-range):

$ curl -sv -H "Range: bytes=-20000" --stderr - -o /dev/null -w 'Response-Length: %{size_download}\nHTTP-Code:%{http_code}\n' <url-redacted> | grep -i 'Length\|Range\|Code'
> range: bytes=-20000
< content-length: 20001
< accept-ranges: bytes
< content-range: bytes -2704-17296/17297
Response-Length: 17297
HTTP-Code:206

content-length should be 17297 (number of returned bytes).
content-range should be "bytes 0-17296/17297"

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.