Further SSL problems with Poetry and a private Gitlab package registry

[marshalc]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 11.3
• Poetry version: 1.1.6

Issue

Closely related to #745 and #1012, I am experiencing problems working with poetry against a private (Gitlab) based package registry, on an internal project. The locked nature of this make this very hard for me to do a public replication of the problem, so apologies whilst I have to be descriptive (if a little vague for some sensitive info).

I have two poetry based python+django projects. The first (atlas-models) I have managed to package as a reusable app, and build and publish (with just poetry) to our internal Gitlab Package Registry. Presently there are two versions of the package in the registry (but I've tried with between 1 & 3 versions there, that doesn't appear to be a factor).

In my second project, when I attempt to add the first package via poetry add atlas-models -vvv, I get the following (abbreviated) output:

➜  ouh-mpages-development-repo git:(master) ✗ poetry add atlas-models -vvv

Using virtualenv: /Users/carl/Projects/pages.oxnet/documentation/ouh-mpages-development/ouh-mpages-development-repo/.venv
pages-oxnet-atlas: 2 packages found for atlas-models *
PyPI: No packages found for atlas-models *
Using version ^0.3.1 for atlas-models

Updating dependencies
Resolving dependencies...
   1: fact: mpages mkdocs build is 0.1.0
   1: derived: mpages mkdocs build
   1: fact: mpages mkdocs build depends on Django (^3.2)
   1: fact: mpages mkdocs build depends on atlas-models (^0.3.1)
   1: selecting mpages mkdocs build (0.1.0)
   1: derived: atlas-models (>=0.3.1,<0.4.0)
   1: derived: Django (>=3.2,<4.0)
pages-oxnet-atlas: 1 packages found for atlas-models >=0.3.1,<0.4.0
PyPI: No packages found for atlas-models >=0.3.1,<0.4.0
   1: fact: atlas-models (0.3.1) depends on Django (>=3.2,<4.0)
   1: fact: atlas-models (0.3.1) depends on cx-oracle (>=8,<9)
   1: selecting atlas-models (0.3.1)
   1: derived: cx-oracle (>=8,<9)
   1: fact: django (3.2) depends on asgiref (>=3.3.2,<4)
   1: fact: django (3.2) depends on pytz (*)
   1: fact: django (3.2) depends on sqlparse (>=0.2.2)
   1: selecting django (3.2)
   1: derived: sqlparse (>=0.2.2)
   1: derived: pytz
   1: derived: asgiref (>=3.3.2,<4)
   1: selecting cx-oracle (8.1.0)
   1: selecting sqlparse (0.4.1)
   1: selecting pytz (2021.1)
   1: selecting asgiref (3.3.4)
   1: Version solving took 0.723 seconds.
   1: Tried 1 solutions.

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals, 5 skipped

  • Installing asgiref (3.3.4): Pending...
  • Installing asgiref (3.3.4): Skipped for the following reason: Already installed
  • Installing pytz (2021.1): Pending...
  • Installing pytz (2021.1): Skipped for the following reason: Already installed
  • Installing sqlparse (0.4.1): Pending...
  • Installing sqlparse (0.4.1): Skipped for the following reason: Already installed
  • Installing cx-oracle (8.1.0): Pending...
  • Installing cx-oracle (8.1.0): Skipped for the following reason: Already installed
  • Installing django (3.2): Pending...
  • Installing django (3.2): Skipped for the following reason: Already installed
  • Installing atlas-models (0.3.1): Pending...
Retrying HTTP request in 0.5 seconds.
Retrying HTTP request in 1.0 seconds.
Retrying HTTP request in 1.5 seconds.
Retrying HTTP request in 2.0 seconds.
Retrying HTTP request in 2.5 seconds.
  • Installing atlas-models (0.3.1): Failed

  SSLError

  HTTPSConnectionPool(host='oxnetcnsc01.oxnet.nhs.uk', port=443): Max retries exceeded with url: /api/v4/projects/202/packages/pypi/files/b278cb68cacfff2d3fd4a3d1a82c6ff12dc192c332e67db76cbdf03a02a10c2a/atlas_models-0.3.1-py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

  at ~/.poetry/lib/poetry/_vendor/py3.9/requests/adapters.py:514 in send
      510│                 raise ProxyError(e, request=request)
      511│ 
      512│             if isinstance(e.reason, _SSLError):
      513│                 # This branch is for urllib3 v1.22 and later.
    → 514│                 raise SSLError(e, request=request)
      515│ 
      516│             raise ConnectionError(e, request=request)
      517│ 
      518│         except ClosedPoolError as e:


Failed to add packages, reverting the pyproject.toml file to its original content.

The pyproject.toml reads as:

name = "MPages MkDocs Build"
version = "0.1.0"
description = "Python / Django application *snip*"
authors = ["Carl Marshall <email-redacted>"]
license = "Apache-2.0"

[tool.poetry.dependencies]
python = "^3.9"
Django = "^3.2"

[tool.poetry.dev-dependencies]

[[tool.poetry.source]]
name = "pages-oxnet-atlas"
url = "https://oxnetcnsc01.oxnet.nhs.uk/api/v4/projects/202/packages/pypi/simple"
secondary = true

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Additionally, poetry config --list -vvv results in:

Loading configuration file /Users/carl/Library/Application Support/pypoetry/config.toml
Loading configuration file /Users/carl/Library/Application Support/pypoetry/auth.toml
cache-dir = "/Users/carl/Library/Caches/pypoetry"
certificates.oxnetcnsc01.cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem"  # None
certificates.pages-oxnet.cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem"  # None
certificates.pages-oxnet-atlas.cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem"  # None
experimental.new-installer = true
installer.parallel = true
repositories.pages-oxnet.url = "https://oxnetcnsc01.oxnet.nhs.uk/api/v4/projects/202/packages/pypi"
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/carl/Library/Caches/pypoetry/virtualenvs

I have also done poetry config http-basic.pages-oxnet user.name token-code for both .pages-oxnet and .pages-oxnet-atlas. This was all pretty much put in place to get the build and publish workflows to succeed. I also added certificates.oxnetcnsc01.cert and http-basic.oxnetcnsc01 as well to ensure I hadn't overlooked anything.

I can see that the process is locating the latest version of the atlas-models package from the gitlab private registry, but is clearly having a problem with downloading the wheel file.

The SSL certificates you can see in the config output above are seemingly valid as they resolved the publish SSL connection issues.

If I try to set the environment variable though (to catch a potential issue with requests as suggested in an earlier bug report), I get a whole host of different errors, i.e.:

• export REQUESTS_CA_BUNDLE=/Users/carl/Projects/pages.oxnet/oxnet-combined.pem followed by
• poetry add atlas-models -vvv results in a much faster set of errors:

Using virtualenv: /Users/carl/Projects/pages.oxnet/documentation/ouh-mpages-development/ouh-mpages-development-repo/.venv
pages-oxnet-atlas: 2 packages found for atlas-models *

  Stack trace:

  23  ~/.poetry/lib/poetry/_vendor/py3.9/clikit/console_application.py:131 in run
       129│             parsed_args = resolved_command.args
       130│ 
     → 131│             status_code = command.handle(parsed_args, io)
       132│         except KeyboardInterrupt:
       133│             status_code = 1

  22  ~/.poetry/lib/poetry/_vendor/py3.9/clikit/api/command/command.py:120 in handle
       118│     def handle(self, args, io):  # type: (Args, IO) -> int
       119│         try:
     → 120│             status_code = self._do_handle(args, io)
       121│         except KeyboardInterrupt:
       122│             if io.is_debug():

  21  ~/.poetry/lib/poetry/_vendor/py3.9/clikit/api/command/command.py:171 in _do_handle
       169│         handler_method = self._config.handler_method
       170│ 
     → 171│         return getattr(handler, handler_method)(args, io, self)
       172│ 
       173│     def __repr__(self):  # type: () -> str

  20  ~/.poetry/lib/poetry/_vendor/py3.9/cleo/commands/command.py:92 in wrap_handle
        90│         self._command = command
        91│ 
     →  92│         return self.handle()
        93│ 
        94│     def handle(self):  # type: () -> Optional[int]

  19  ~/.poetry/lib/poetry/console/commands/add.py:106 in handle
       104│             return 0
       105│ 
     → 106│         requirements = self._determine_requirements(
       107│             packages,
       108│             allow_prereleases=self.option("allow-prereleases"),

  18  ~/.poetry/lib/poetry/console/commands/init.py:328 in _determine_requirements
       326│             elif "version" not in requirement:
       327│                 # determine the best version automatically
     → 328│                 name, version = self._find_best_version_for_package(
       329│                     requirement["name"],
       330│                     allow_prereleases=allow_prereleases,

  17  ~/.poetry/lib/poetry/console/commands/init.py:361 in _find_best_version_for_package
       359│ 
       360│         selector = VersionSelector(self._get_pool())
     → 361│         package = selector.find_best_candidate(
       362│             name, required_version, allow_prereleases=allow_prereleases, source=source
       363│         )

  16  ~/.poetry/lib/poetry/version/version_selector.py:32 in find_best_candidate
       30│             },
       31│         )
     → 32│         candidates = self._pool.find_packages(dependency)
       33│         only_prereleases = all([c.version.is_prerelease() for c in candidates])
       34│ 

  15  ~/.poetry/lib/poetry/repositories/pool.py:165 in find_packages
       163│         packages = []
       164│         for repo in self._repositories:
     → 165│             packages += repo.find_packages(dependency)
       166│ 
       167│         return packages

  14  ~/.poetry/lib/poetry/repositories/pypi_repository.py:101 in find_packages
        99│ 
       100│         try:
     → 101│             info = self.get_package_info(dependency.name)
       102│         except PackageNotFound:
       103│             self._log(

  13  ~/.poetry/lib/poetry/repositories/pypi_repository.py:202 in get_package_info
       200│             return self._get_package_info(name)
       201│ 
     → 202│         return self._cache.store("packages").remember_forever(
       203│             name, lambda: self._get_package_info(name)
       204│         )

  12  ~/.poetry/lib/poetry/_vendor/py3.9/cachy/repository.py:174 in remember_forever
       172│             return val
       173│ 
     → 174│         val = value(callback)
       175│ 
       176│         self.forever(key, val)

  11  ~/.poetry/lib/poetry/_vendor/py3.9/cachy/helpers.py:6 in value
       4│ def value(val):
       5│     if callable(val):
     → 6│         return val()
       7│ 
       8│     return val

  10  ~/.poetry/lib/poetry/repositories/pypi_repository.py:203 in <lambda>
       201│ 
       202│         return self._cache.store("packages").remember_forever(
     → 203│             name, lambda: self._get_package_info(name)
       204│         )
       205│ 

   9  ~/.poetry/lib/poetry/repositories/pypi_repository.py:207 in _get_package_info
       205│ 
       206│     def _get_package_info(self, name):  # type: (str) -> dict
     → 207│         data = self._get("pypi/{}/json".format(name))
       208│         if data is None:
       209│             raise PackageNotFound("Package [{}] not found.".format(name))

   8  ~/.poetry/lib/poetry/repositories/pypi_repository.py:315 in _get
       313│     def _get(self, endpoint):  # type: (str) -> Union[dict, None]
       314│         try:
     → 315│             json_response = self.session.get(self._base_url + endpoint)
       316│         except requests.exceptions.TooManyRedirects:
       317│             # Cache control redirect loop.

   7  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:555 in get
       553│ 
       554│         kwargs.setdefault('allow_redirects', True)
     → 555│         return self.request('GET', url, **kwargs)
       556│ 
       557│     def options(self, url, **kwargs):

   6  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:542 in request
       540│         }
       541│         send_kwargs.update(settings)
     → 542│         resp = self.send(prep, **send_kwargs)
       543│ 
       544│         return resp

   5  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:677 in send
       675│             # Redirect resolving generator.
       676│             gen = self.resolve_redirects(r, request, **kwargs)
     → 677│             history = [resp for resp in gen]
       678│         else:
       679│             history = []

   4  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:677 in <listcomp>
       675│             # Redirect resolving generator.
       676│             gen = self.resolve_redirects(r, request, **kwargs)
     → 677│             history = [resp for resp in gen]
       678│         else:
       679│             history = []

   3  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:237 in resolve_redirects
       235│             else:
       236│ 
     → 237│                 resp = self.send(
       238│                     req,
       239│                     stream=stream,

   2  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:655 in send
       653│ 
       654│         # Send the request
     → 655│         r = adapter.send(request, **kwargs)
       656│ 
       657│         # Total elapsed time of the request (approximately)

   1  ~/.poetry/lib/poetry/_vendor/py3.9/cachecontrol/adapter.py:53 in send
        51│             request.headers.update(self.controller.conditional_headers(request))
        52│ 
     →  53│         resp = super(CacheControlAdapter, self).send(request, **kw)
        54│ 
        55│         return resp

  SSLError

  HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/atlas-models/json/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

  at ~/.poetry/lib/poetry/_vendor/py3.9/requests/adapters.py:514 in send
      510│                 raise ProxyError(e, request=request)
      511│ 
      512│             if isinstance(e.reason, _SSLError):
      513│                 # This branch is for urllib3 v1.22 and later.
    → 514│                 raise SSLError(e, request=request)
      515│ 
      516│             raise ConnectionError(e, request=request)
      517│ 
      518│         except ClosedPoolError as e:

When I unset REQUESTS_CA_BUNDLE and retry the add command, we're back to the original error outputs and timeout.

I am now at a loss of what else to try here. Something appears to be partly broken in the process of adding my package from the internal registry, and this is now stalling my whole development plan. I don't really want to have to remove poetry from this setup, but I do note there's very little Gitlab recognition of Poetry usage (none in their own documentation) and most examples I'm finding at either general pip or GitHub solutions.

Please let me know if there's any other practical tests, or configuration I can attempt, or if this is enough information on which to diagnose a bug.

Thank you for your time and help.

[abn]

The problem seems to be missing cert in your system bundle and when you set the env var the default bundles being disabled.

Does setting a custom cert help here? There was also a fix (on master) related to this config.

Also you should consider adding via poetry add --source name package.

[marshalc]

I'm not sure I'm following your logic on the missing cert, but I also think I'm missing some understanding on how poetry is expecting certificates to be presented to it. Working this through...

The .pem is a two certificate file with intermediary and root CA certs inside it, and set as .cert for three possible configurations (though I think only the pages-oxnet-atlas config is the one in use here`).

It is the same .pem that unlocks the SSL chain for the publish process (which is working) - so I'm not clear as to why it isn't missing for one process, but is for another?

AFAIK there's no client certificate needed to authenticate to the Gitlab (I've had no git related SSL issues, nor regular browser related login issues), so .client-cert would seem to not be relevant. Just to be sure though, I have tried setting .client-cert for all three configurations, i.e.:

certificates.oxnetcnsc01.cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem"  # None
certificates.oxnetcnsc01.client-cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem"  # None
certificates.pages-oxnet.cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem"  # None
certificates.pages-oxnet.client-cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem"  # None
certificates.pages-oxnet-atlas.cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem"  # None
certificates.pages-oxnet-atlas.client-cert = "/Users/carl/Projects/pages.oxnet/oxnet-combined.pem"  # None

... and it broke things in a different way (which I kind of expected since those CA certs aren't client certs)

For those searching, that break looks like:

✗ poetry add --source pages-oxnet-atlas atlas-models -vvv
Using virtualenv: /Users/carl/Projects/pages.oxnet/documentation/ouh-mpages-development/ouh-mpages-development-repo/.venv

  Stack trace:

  15  ~/.poetry/lib/poetry/_vendor/py3.9/clikit/console_application.py:131 in run
       129│             parsed_args = resolved_command.args
       130│ 
     → 131│             status_code = command.handle(parsed_args, io)
       132│         except KeyboardInterrupt:
       133│             status_code = 1

  14  ~/.poetry/lib/poetry/_vendor/py3.9/clikit/api/command/command.py:120 in handle
       118│     def handle(self, args, io):  # type: (Args, IO) -> int
       119│         try:
     → 120│             status_code = self._do_handle(args, io)
       121│         except KeyboardInterrupt:
       122│             if io.is_debug():

  13  ~/.poetry/lib/poetry/_vendor/py3.9/clikit/api/command/command.py:171 in _do_handle
       169│         handler_method = self._config.handler_method
       170│ 
     → 171│         return getattr(handler, handler_method)(args, io, self)
       172│ 
       173│     def __repr__(self):  # type: () -> str

  12  ~/.poetry/lib/poetry/_vendor/py3.9/cleo/commands/command.py:92 in wrap_handle
        90│         self._command = command
        91│ 
     →  92│         return self.handle()
        93│ 
        94│     def handle(self):  # type: () -> Optional[int]

  11  ~/.poetry/lib/poetry/console/commands/add.py:106 in handle
       104│             return 0
       105│ 
     → 106│         requirements = self._determine_requirements(
       107│             packages,
       108│             allow_prereleases=self.option("allow-prereleases"),

  10  ~/.poetry/lib/poetry/console/commands/init.py:328 in _determine_requirements
       326│             elif "version" not in requirement:
       327│                 # determine the best version automatically
     → 328│                 name, version = self._find_best_version_for_package(
       329│                     requirement["name"],
       330│                     allow_prereleases=allow_prereleases,

   9  ~/.poetry/lib/poetry/console/commands/init.py:361 in _find_best_version_for_package
       359│ 
       360│         selector = VersionSelector(self._get_pool())
     → 361│         package = selector.find_best_candidate(
       362│             name, required_version, allow_prereleases=allow_prereleases, source=source
       363│         )

   8  ~/.poetry/lib/poetry/version/version_selector.py:32 in find_best_candidate
       30│             },
       31│         )
     → 32│         candidates = self._pool.find_packages(dependency)
       33│         only_prereleases = all([c.version.is_prerelease() for c in candidates])
       34│ 

   7  ~/.poetry/lib/poetry/repositories/pool.py:165 in find_packages
       163│         packages = []
       164│         for repo in self._repositories:
     → 165│             packages += repo.find_packages(dependency)
       166│ 
       167│         return packages

   6  ~/.poetry/lib/poetry/repositories/legacy_repository.py:264 in find_packages
       262│             versions = self._cache.store("matches").get(key)
       263│         else:
     → 264│             page = self._get("/{}/".format(dependency.name.replace(".", "-")))
       265│             if page is None:
       266│                 return []

   5  ~/.poetry/lib/poetry/repositories/legacy_repository.py:388 in _get
       386│         url = self._url + endpoint
       387│         try:
     → 388│             response = self.session.get(url)
       389│             if response.status_code == 404:
       390│                 return

   4  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:555 in get
       553│ 
       554│         kwargs.setdefault('allow_redirects', True)
     → 555│         return self.request('GET', url, **kwargs)
       556│ 
       557│     def options(self, url, **kwargs):

   3  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:542 in request
       540│         }
       541│         send_kwargs.update(settings)
     → 542│         resp = self.send(prep, **send_kwargs)
       543│ 
       544│         return resp

   2  ~/.poetry/lib/poetry/_vendor/py3.9/requests/sessions.py:655 in send
       653│ 
       654│         # Send the request
     → 655│         r = adapter.send(request, **kwargs)
       656│ 
       657│         # Total elapsed time of the request (approximately)

   1  ~/.poetry/lib/poetry/_vendor/py3.9/cachecontrol/adapter.py:53 in send
        51│             request.headers.update(self.controller.conditional_headers(request))
        52│ 
     →  53│         resp = super(CacheControlAdapter, self).send(request, **kw)
        54│ 
        55│         return resp

  SSLError

  HTTPSConnectionPool(host='oxnetcnsc01.oxnet.nhs.uk', port=443): Max retries exceeded with url: /api/v4/projects/202/packages/pypi/simple/atlas-models/ (Caused by SSLError(SSLError(9, '[SSL] PEM lib (_ssl.c:4048)')))

  at ~/.poetry/lib/poetry/_vendor/py3.9/requests/adapters.py:514 in send
      510│                 raise ProxyError(e, request=request)
      511│ 
      512│             if isinstance(e.reason, _SSLError):
      513│                 # This branch is for urllib3 v1.22 and later.
    → 514│                 raise SSLError(e, request=request)
      515│ 
      516│             raise ConnectionError(e, request=request)
      517│ 
      518│         except ClosedPoolError as e:

By removing the lines from auth.toml, we're back to the original error again:

*snip*

  • Installing django (3.2): Skipped for the following reason: Already installed
  • Installing atlas-models (0.3.1): Pending...
Retrying HTTP request in 0.5 seconds.
Retrying HTTP request in 1.0 seconds.
Retrying HTTP request in 1.5 seconds.
Retrying HTTP request in 2.0 seconds.
Retrying HTTP request in 2.5 seconds.
  • Installing atlas-models (0.3.1): Failed

  SSLError

  HTTPSConnectionPool(host='oxnetcnsc01.oxnet.nhs.uk', port=443): Max retries exceeded with url: /api/v4/projects/202/packages/pypi/files/b278cb68cacfff2d3fd4a3d1a82c6ff12dc192c332e67db76cbdf03a02a10c2a/atlas_models-0.3.1-py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

  at ~/.poetry/lib/poetry/_vendor/py3.9/requests/adapters.py:514 in send
      510│                 raise ProxyError(e, request=request)
      511│ 
      512│             if isinstance(e.reason, _SSLError):
      513│                 # This branch is for urllib3 v1.22 and later.
    → 514│                 raise SSLError(e, request=request)
      515│ 
      516│             raise ConnectionError(e, request=request)
      517│ 
      518│         except ClosedPoolError as e:


Failed to add packages, reverting the pyproject.toml file to its original content.

So, looking at that and at https://stackoverflow.com/questions/51925384/unable-to-get-local-issuer-certificate-when-using-requests-in-python, I've added the root and inter ca's to the certifi cacert.pem, confirmed that they're being seen by certifi... but that makes no difference to poetry, same error (and I note that that there's no certifi in the local venv, so I'm guessing that this isn't really being used).

Stepping back and trying the method Gitlab help suggests (i.e. pip install atlas-models --no-deps --index-url https://carl.marshall:<token>@oxnetcnsc01.oxnet.nhs.uk/api/v4/projects/202/packages/pypi/simple), and we have similar issues:

Looking in indexes: https://carl.marshall:****@oxnetcnsc01.oxnet.nhs.uk/api/v4/projects/202/packages/pypi/simple
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))': /api/v4/projects/202/packages/pypi/simple/atlas-models/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))': /api/v4/projects/202/packages/pypi/simple/atlas-models/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))': /api/v4/projects/202/packages/pypi/simple/atlas-models/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))': /api/v4/projects/202/packages/pypi/simple/atlas-models/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))': /api/v4/projects/202/packages/pypi/simple/atlas-models/
Could not fetch URL https://carl.marshall:****@oxnetcnsc01.oxnet.nhs.uk/api/v4/projects/202/packages/pypi/simple/atlas-models/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='oxnetcnsc01.oxnet.nhs.uk', port=443): Max retries exceeded with url: /api/v4/projects/202/packages/pypi/simple/atlas-models/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))) - skipping
ERROR: Could not find a version that satisfies the requirement atlas-models
ERROR: No matching distribution found for atlas-models

If I visit the URL in a browser (https://oxnetcnsc01.oxnet.nhs.uk/api/v4/projects/202/packages/pypi/simple/atlas-models/ - behind a firewall so this won't work for external/internet access) I am prompted for my username + token login details, and then presented with a list of packages (.tar.gz and .whl files for atlas-models) and if I click on the wheel link, then it downloads the wheel file just fine. Looking in the browser I can see a valid certificate chain for the custom root, inter, and server certificates.

So if I'm understanding your first line @abn - I need to figure out which certificate bundle that python is using because that appears to be the weak link here; it also seems that poetry is ignoring the certificates I'm setting as custom for this add process, but is using them for the publish process. Investigation continues, though if anyone has any helpful pointers for a python installed via homebrew on macOS setup, please say!

[marshalc]

Fixed it!

https://stackoverflow.com/a/57795811 provided the clue here. Following on from the certifi install I did for the base python3 homebrew install, adding:

CERT_PATH=$(python3 -m certifi)
export SSL_CERT_FILE=${CERT_PATH}
export REQUESTS_CA_BUNDLE=${CERT_PATH}

to my ~/zshrc and restarting the process, meant that when I next ran poetry add --source pages-oxnet-atlas atlas-models -vvv, we were in like lightning with:

*snip*
  • Installing django (3.2): Skipped for the following reason: Already installed
  • Installing atlas-models (0.3.1): Downloading... 100%
  • Installing atlas-models (0.3.1): Installing...
  • Installing atlas-models (0.3.1)

I'm still confused as to why publish worked with the certificate config in poetry, but add didn't, but at least with this fix in I can carry on working.

[nylocx]

I hope it is ok to piggyback on this ticket.
I have a pretty simple project and a self signed repository.
I did the following:

poetry config repositories.myrepo "https://my.repo.tld/simple/"
poetry config certificates.myrepo.cert "/absolute/path/to/my.repo.tld.pem"

Than I added a section in my pyproject.toml

[[tool.poetry.source]]
name = "myrepo"
url = "https://my.repo.tld/simple/"
secondary = true

If I run a poetry add <any-package-that-is-on-pypi>
I get:

HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/networkx/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')))

If I just remove the tool.poetry.source block from my pyproject.toml everything is back to normal and works.
I'm on Poetry 1.1.6

It looks like setting any certificate in the config overrides the my system certificate and the workaround from marshalc does not work for me setting SSL_CERT_FILE or REQUESTS_CA_BUNDLE to a valid certificate bundle does not change the behavior.
I'm happy for any hint, as this currently blocks my progress.

[dabla]

I'm experiencing the same issue on poetry 1.1.8 but I have a patch to solve this. The problem is that by default, the requests library doesn't retry when a connection fails, and the custom poetry CacheControlAdapter also doesn't pass any retry parameters. Once the patch is done, I'll ask a merge request to fix this issue. In our case this issue only arises on Windows which is behind an enterprise proxy, but not on Linux, but as most developers work on Windows machines, we needed to fix this issue. We didn't experience this issue with poetry 1.0.10 though.

[gazpachoking]

Okay, I was having this same issue, and have found exactly what the cause is (at least in my case.) When determining what repository is associated with a given url, it compares the netloc of the registered custom repository to the netloc of the given url to fetch.

poetry/src/poetry/utils/authenticator.py

Line 397 in bf0dbc0

if repository.netloc == parsed_url.netloc:

The problem is that gitlab tells you to set up your custom repository url including an username and password, e.g. https://__toke__:[email protected]/api/v4/projects/5/packages/pypi/simple, but the link for the wheel it provides does not include the username and password, thus causing the netloc for the weel download to not match the repository netloc. poetry then doesn't use the custom certificate when validating that url.

I think the proper solution would be to adjust the poetry authentcator utility to only compare the hostname and port, rather than the whole netloc when determining what certs to use.

[R1tschY]

I also had this problem. In the end I fixed it for me with removing the cert repository configuration in the auth.toml:

poetry config certificates.myrepo.cert --unset

Additional I used the fix from marshalc to set the explicit certificate store and added the missing certificate to the CERT_PATH file:

CERT_PATH=$(python -c "import certifi; print(certifi.where())")
echo PATH/TO/CA.crt >> $CERT_PATH
export REQUESTS_CA_BUNDLE=${CERT_PATH}
export SSL_CERT_FILE=${CERT_PATH}