HTTPX and Celery \ GEvent Issue

[pg1671]

I have a gevent based celery worker that will not start if httpx is installed. My code does not use HTTPX, it was installed as a dependancy of openai.

If I pip uninstall httpx my worker runs again.

I am using Python 3.11 and the latest celery.

I also posted the issue here: celery/celery#9032

The main issue is I can't find any indication of the conflict as the celery worker just dies with nothing in the logs or system journal.

I am assuming some other package is dynamically using httpx when it is available but leads to an issue under gevent.

Is there a way to debug the package loading process to see what is trying to use httpx?

[pg1671]

This looks to be caused by the httpx dependancy on trio.

If I run the code:
from gevent.monkey import patch_all; patch_all()
import gevent
import httpx

def worker(n):
print(n)

if name == 'main':
jobs = [gevent.spawn(worker, job_no) for job_no in range(10)]
gevent.joinall(jobs)

I get:
File "/tests/gevent_test.py", line 3, in
import httpx
File "/work/env3.11/lib/python3.11/site-packages/httpx/init.py", line 2, in
from ._api import delete, get, head, options, patch, post, put, request, stream
File "/work/env3.11/lib/python3.11/site-packages/httpx/_api.py", line 6, in
from ._client import Client
File "/work/env3.11/lib/python3.11/site-packages/httpx/_client.py", line 32, in
from ._transports.default import AsyncHTTPTransport, HTTPTransport
File "/work/env3.11/lib/python3.11/site-packages/httpx/_transports/default.py", line 32, in
import httpcore
File "/work/env3.11/lib/python3.11/site-packages/httpcore/init.py", line 1, in
from ._api import request, stream
File "/work/env3.11/lib/python3.11/site-packages/httpcore/_api.py", line 5, in
from ._sync.connection_pool import ConnectionPool
File "/work/env3.11/lib/python3.11/site-packages/httpcore/_sync/init.py", line 1, in
from .connection import HTTPConnection
File "/work/env3.11/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 12, in
from .._synchronization import Lock
File "/work/env3.11/lib/python3.11/site-packages/httpcore/_synchronization.py", line 11, in
import trio
File "/work/env3.11/lib/python3.11/site-packages/trio/init.py", line 23, in
from ._core import TASK_STATUS_IGNORED as TASK_STATUS_IGNORED # isort: split
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/work/env3.11/lib/python3.11/site-packages/trio/_core/init.py", line 21, in
from ._local import RunVar, RunVarToken
File "/work/env3.11/lib/python3.11/site-packages/trio/_core/_local.py", line 9, in
from . import _run
File "/work/env3.11/lib/python3.11/site-packages/trio/_core/_run.py", line 2840, in
raise NotImplementedError("unsupported platform")
NotImplementedError: unsupported platform

[tomchristie]

To clarify... we don't have a dependency on trio, although it is optional and will be in imported if installed.

Could you migrate this issue to the httpcore repo, and we'll deal with it at the source?

[pg1671]

Thanks tom .. The issue I am experiencing only happens when httpx is installed. If I removed httpx and left httpcore everything was fine.

I did further testing yesterday and the underlying cause was trio.

python-trio/trio#3013

My application does not use either httpx or trio. Httpx is a dependancy of the openai > 1.0.0 package. My code that runs under gevent (via celery gevent worker) does not call any functions that use httpx or trio but having them installed causes the problem where the worker will not start.

celery/celery#9032

I was just pinning openai to the versions prior to using httpx for now. But found the best option was just to uninstall trio which was the root cause.

I am hoping to use httpx under gevent in the future so I can make http 2.0 calls and to call the AI libraries that use HTTPX under gevent.

Not sure If I will run into other issues then.

Let me know if you still want me to move this to httpcore.

[tomchristie]

Okay, here's the issue in httpcore... encode/httpcore#946 ...there's an import try ... catch that can be fixed up to resolve this. You're welcome to tackle a PR for it if you'd like.

[see-ann]

Has this issue been resolved?

[tomchristie]

Thanks @see-ann, yes. Resolved in httpcore 1.0.6. encode/httpcore#955