Pyarrow and asyncio

[SaeedAr]

I'm trying to use pyarrow in an asynchronous python code-base for processing stream of batches of data.
I found a sample here that basically wraps pyarrow RecordBatchStreamWriter with asyncio sleeps to make an asynchronous behavior. Is this the best possible way to use pyarrow streams asynchronously?

I also found some experimental support for asyncio in pyarrow.plasma here which again by async wait. As I understood IO write/read functions in pyarrow are all synchronous and blocking. Is there any plan for asyncio support in pyarrow?

[wesm]

I don't see why not. Would you like to open some JIRA issues proposing changes to the library to provide asyncio support?

[thomafred]

Hi @SaeedAr, did you find a working solution?

[gatesn]

I put together a Gist with a rough (haven't fully tested error cases) implementation of an AsyncMessageReader. This should be enough to build out an AsyncRecordBatchReader etc.

https://gist.github.com/gatesn/86462c33d765b0fc63d7bb88529204d0