Using Flight RPC Calls as a table

[nathanielc]

I would like to consume the results of a Flight RPC call as a table within a datafusion context. So far I have not found a built-in way to do this. Is there an example out there of consuming a Flight RPC stream as a TableProvider?

Looking through the available types I am expecting I would need to do the following:

1. Implement TableProvider trait that builds an ExecutionPlan when scan is called.
2. Implement the ExecutionPlan in terms of the StreamingTableExec type by implementing the PartitionStream trait on the returned stream from the Flight RPC call.

Does this sound like what is expected? Or is there a more direct path to exposing a stream of RecordBatches from Flight RPC as a table in datafusion? Thanks in advance.

[backkem]

There is a Flight SQL version of this idea here. You may be able to simplify it to a regular Flight RPC implementation.

If you do manage to make it work. Maybe there's a place for it in datafusion-contrib/datafusion-table-provider

[nathanielc]

Excellent, a quick look show that will be very helpful. I'll see if there is a generic way to federate the RPC flavor. Might need to abstract away the FlightDescriptors etc so we get proper push down filters and projections. Thanks

[backkem]

In case you're interested in full blown remote querying, that's what datafusion-contrib/datafusion-federation itself does. Instead of filter/projection pushdown it tries to 'push down's entire (sub-)queries by turning them back to SQL. There is an example SQL Flight server implementation as well.