Make list backfills endpoint use asyncio #44208
Conversation
dstandish
changed the title
There was a problem hiding this comment.
Looks nice.
Just a few small suggestions.
Also I realized that we cannot have 'lazy' loading of some attributes at serialization time (when using pydantic), because this will be 'uncontroled async I/O'.
I don't know if you are encountering this on the Backfill model, but basically it means that everything needs to be eager loaded before, so pydantic does not emit uncontrolled request afterwards. (and unfortunatly we cannot await BackfillResponse.model_validate)
Also just to mention that there is a stream API to iterate over multiple results. That could be interesting to see if this can help us here.
Given that sometimes we don't want to apply any filters, it makes sense to make the param optional. I also fix the typing on `paginated_select`.
This is a sort of hello world for having an route implemented using asyncio.
dstandish
force-pushed
the
poc-async-route-fastapi
branch
from
74cbd14
to
1efe639
Compare
airflow/settings.py
Outdated
| @@ -498,7 +498,7 @@ def configure_orm(disable_connection_pool=False, pool_class=None): | |||
|
|
|||
| engine = create_engine(SQL_ALCHEMY_CONN, connect_args=connect_args, **engine_args, future=True) | |||
| async_engine = create_async_engine(SQL_ALCHEMY_CONN_ASYNC, future=True) | |||
| create_async_session = sessionmaker( | |||
| session_maker_async = sessionmaker( | |||
There was a problem hiding this comment.
Your name is more explicit, and I understand why legacy names can feel confusing and this one could be better. (We call a session_maker Session oO => but that's actually a factory for session, so from the client code perspective we are constructing a session and we think we are manipulating a Session class for instantiation, that's especially true for the threadlocal registry scoped_session, just create a Session but actually things happen under the hood, we are not just instantiating a session and we don't ask people to explicitely use a factory, just the normal Session, that's how sqlalchemy docs and many other documents online recommend to do it.)
Anyway, for consistency, I recommend using AsyncSession. (same way as Session and NonScopedSession above).
In the user code this it just becomes session = AsyncSession()
There was a problem hiding this comment.
to clarify, with this
that's how sqlalchemy docs and many other documents online recommend to do it
you are saying that the SA docs recommend doing it by storing session factory as AsyncSession, like we did with Session?
There was a problem hiding this comment.
oh, yeah, here i guess https://docs.sqlalchemy.org/en/20/orm/session_api.html#session-and-sessionmaker
This is a sort of hello world / proof of concept for having an route implemented using asyncio.
Depends on #44226