Skip to content

Installing Prefect using Poetry on an M1 fails #6310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
4 tasks done
lebovic opened this issue Aug 8, 2022 · 5 comments
Closed
4 tasks done

Installing Prefect using Poetry on an M1 fails #6310

lebovic opened this issue Aug 8, 2022 · 5 comments
Assignees
Labels
bug Something isn't working great writeup This is a wonderful example of our standards

Comments

@lebovic
Copy link

lebovic commented Aug 8, 2022

First check

  • I added a descriptive title to this issue.
  • I used the GitHub search to find a similar issue and didn't find it.
  • I searched the Prefect documentation for this issue.
  • I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

Installing Prefect on an M1 with Poetry skips the SQLAlchemy dependency greenlet.

It seems like this is a known issue with SQLAlchemy + Greenlet (see context) – and explicitly adding sqlalchemy[asyncio] as is done in Prefect's requirements.txt fixes with pip – but a Poetry install of Prefect on an M1 still fails.

Reproduction

On an M1 machine:

  1. poetry new test-project
  2. poetry add prefect

In a new file, pipeline.py:

from prefect import flow


@flow
def my_favorite_function():
    print("What is your favorite number?")
    return 42

print(my_favorite_function())

poetry run python3 pipeline.py

Error

Traceback (most recent call last):
  File "pipeline.py", line 9, in <module>
    print(my_favorite_function())
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/flows.py", line 390, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/engine.py", line 152, in enter_flow_run_engine_from_flow_call
    return anyio.run(begin_run)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/client.py", line 102, in with_injected_client
    async with client_context as client:
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/client.py", line 1940, in __aenter__
    self._ephemeral_lifespan = await self._exit_stack.enter_async_context(
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/contextlib.py", line 568, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/contextlib.py", line 171, in __aenter__
    return await self.gen.__anext__()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/client.py", line 186, in app_lifespan_context
    await context.__aenter__()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/asgi_lifespan/_manager.py", line 92, in __aenter__
    await self._exit_stack.aclose()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/contextlib.py", line 621, in aclose
    await self.__aexit__(None, None, None)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/contextlib.py", line 679, in __aexit__
    raise exc_details[1]
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/contextlib.py", line 662, in __aexit__
    cb_suppress = await cb(*exc_details)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/asgi_lifespan/_concurrency/asyncio.py", line 80, in __aexit__
    await self.task
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/asgi_lifespan/_manager.py", line 90, in __aenter__
    await self.startup()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/asgi_lifespan/_manager.py", line 36, in startup
    raise self._app_exception
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/asgi_lifespan/_concurrency/asyncio.py", line 63, in run_and_silence_cancelled
    await self.coroutine()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/asgi_lifespan/_manager.py", line 64, in run_app
    await self.app(scope, self.receive, self.send)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/fastapi/applications.py", line 269, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/starlette/middleware/errors.py", line 149, in __call__
    await self.app(scope, receive, send)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/starlette/middleware/cors.py", line 76, in __call__
    await self.app(scope, receive, send)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/starlette/exceptions.py", line 69, in __call__
    await self.app(scope, receive, send)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/starlette/routing.py", line 659, in __call__
    await self.lifespan(scope, receive, send)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/starlette/routing.py", line 635, in lifespan
    async with self.lifespan_context(app):
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/starlette/routing.py", line 530, in __aenter__
    await self._router.startup()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/starlette/routing.py", line 612, in startup
    await handler()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/orion/api/server.py", line 271, in run_migrations
    await db.create_db()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/orion/database/interface.py", line 53, in create_db
    await self.run_migrations_upgrade()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/orion/database/interface.py", line 61, in run_migrations_upgrade
    await run_sync_in_worker_thread(alembic_upgrade)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 56, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/orion/database/alembic_commands.py", line 29, in alembic_upgrade
    alembic.command.upgrade(alembic_config(), revision, sql=dry_run)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/alembic/command.py", line 322, in upgrade
    script.run_env()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/alembic/script/base.py", line 569, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/orion/database/migrations/env.py", line 98, in <module>
    apply_migrations()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 189, in wrapper
    return run_async_from_worker_thread(async_fn, *args, **kwargs)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 136, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/orion/database/migrations/env.py", line 88, in apply_migrations
    engine = await db_interface.engine()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/orion/database/interface.py", line 71, in engine
    engine = await self.database_config.engine()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/prefect/orion/database/configurations.py", line 215, in engine
    engine = create_async_engine(self.connection_url, echo=self.echo, **kwargs)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/sqlalchemy/ext/asyncio/engine.py", line 43, in create_async_engine
    sync_engine = _create_engine(*arg, **kw)
  File "<string>", line 2, in create_engine
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py", line 309, in warned
    return fn(*args, **kwargs)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/sqlalchemy/engine/create.py", line 660, in create_engine
    event.listen(pool, "connect", on_connect)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/sqlalchemy/event/api.py", line 115, in listen
    _event_key(target, identifier, fn).listen(*args, **kw)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/sqlalchemy/event/registry.py", line 232, in listen
    self.dispatch_target.dispatch._listen(self, *args, **kw)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/sqlalchemy/pool/events.py", line 69, in _listen
    event_key.base_listen(**kw)
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/sqlalchemy/event/registry.py", line 270, in base_listen
    for_modify._set_asyncio()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/sqlalchemy/event/attr.py", line 274, in _set_asyncio
    self._exec_once_mutex = AsyncAdaptedLock()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/sqlalchemy/util/concurrency.py", line 67, in AsyncAdaptedLock
    _not_implemented()
  File "/Users/lebovic/Library/Caches/pypoetry/virtualenvs/test-project-GQM6Mw9q-py3.8/lib/python3.8/site-packages/sqlalchemy/util/concurrency.py", line 47, in _not_implemented
    raise ValueError(
ValueError: the greenlet library is required to use this function. No module named 'greenlet'

Versions

Version:             2.0.3
API version:         0.8.0
Python version:      3.8.9
Git commit:          2f1cf4ac
Built:               Fri, Aug 5, 2022 3:57 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.36.0

Additional context

After manually adding greenlet (poetry add greenlet) this works fine! It looks like Poetry isn't picking up the extra in sqlalchemy[asyncio].

This might be outside of fair game for a Prefect issue, because it's an issue with a Prefect dependency – and only while using Poetry. Opening this issue because @madkinsz introduced me to Poetry :)

There are closed issues on SQLAlchemy (sqlalchemy/sqlalchemy#7714) and greenlet (python-greenlet/greenlet#234).

@lebovic lebovic added bug Something isn't working status:triage labels Aug 8, 2022
@zanieb zanieb added great writeup This is a wonderful example of our standards priority:medium and removed status:triage labels Aug 8, 2022
@zanieb
Copy link
Contributor

zanieb commented Aug 8, 2022

Hey! Nice to see you :)

There's another installation error with Poetry at #6278 — sounds like we need to dedicate some attention to compatibility with it.

@zanieb zanieb self-assigned this Aug 8, 2022
@zanieb
Copy link
Contributor

zanieb commented Aug 8, 2022

Hm. Our published wheel explicitly includes the [asyncio] extra for sqlalchemy. Additionally, the extra is captured in my poetry.lock file

(some content omitted)

[[package]]
name = "prefect"
version = "2.0.3"

[package.dependencies]
sqlalchemy = {version = ">=1.4.20,<1.4.33 || >1.4.33", extras = ["asyncio"]}


[[package]]
name = "sqlalchemy"

[package.dependencies]
greenlet = {version = "!=0.4.17", markers = "python_version >= \"3\" and (platform_machine == \"aarch64\" or platform_machine == \"ppc64le\" or platform_machine == \"x86_64\" or platform_machine == \"amd64\" or platform_machine == \"AMD64\" or platform_machine == \"win32\" or platform_machine == \"WIN32\")"}

[package.extras]
asyncio = ["greenlet (!=0.4.17)"]

and poetry show includes greenlet as a dependency.

❯ poetry show sqlalchemy
name         : sqlalchemy
version      : 1.4.40
description  : Database Abstraction Library

dependencies
 - greenlet !=0.4.17

If I created a new poetry shell for isolation, I could reproduce your issue. Interestingly, pip does not show the dependency:

❯ pip show sqlalchemy
Name: SQLAlchemy
Version: 1.4.40
Summary: Database Abstraction Library
Home-page: https://www.sqlalchemy.org
Author: Mike Bayer
Author-email: [email protected]
License: MIT
Location: /Users/mz/Library/Caches/pypoetry/virtualenvs/test--e9fUeN2-py3.9/lib/python3.9/site-packages
Requires: 
Required-by: alembic, prefect

but if I run this command in my working conda environment, it does:

❯ pip show sqlalchemy
Name: SQLAlchemy
Version: 1.4.40
Summary: Database Abstraction Library
Home-page: https://www.sqlalchemy.org
Author: Mike Bayer
Author-email: [email protected]
License: MIT
Location: /opt/homebrew/Caskroom/miniconda/base/envs/orion-dev-38/lib/python3.8/site-packages
Requires: greenlet
Required-by: alembic, prefect

This seems like a poetry bug.

@lebovic
Copy link
Author

lebovic commented Aug 8, 2022

Thanks for looking into this! Awesome to see you here.

That matches what I'm seeing on my Poetry environment too. greenlet=1.1.2 is selected, but silently fails to install ¯\_(ツ)_/¯

Does indeed seem like a Poetry bug.

@github-actions
Copy link
Contributor

This issue is stale because it has been open 30 days with no activity. To keep this issue open remove stale label or comment.

@github-actions
Copy link
Contributor

This issue was closed because it has been stale for 7 days with no activity. If this issue is important or you have more to add feel free to re-open it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working great writeup This is a wonderful example of our standards
Projects
None yet
Development

No branches or pull requests

3 participants