Feature: Databricks SQL Database/Lakehouse support

[jtrangel]

Would love to see a connector or integration over Databricks. Not sure if it will be tricky since some of the data objects (tables/views) in Databricks aren't actually within the databricks filesystem. Others are just referenced externally from object storage (ADLS/S3).

[wwwy3y3]

Hi @jtrangel , thanks for the feature request.

... Not sure if it will be tricky since some of the data objects (tables/views) in Databricks aren't actually within the databricks filesystem. Others are just referenced externally from object storage (ADLS/S3).

For object storage, you can use DuckDB (https://docs.getwren.ai/guide/connect/duckdb) to query files (ex: csv, parquet, json).

Would love to see a connector or integration over Databricks. ...

Additionally, we're actually working on using the ibis-project to connect to the datasource. The concept is that WrenAI generates ibis-executable SQL, which is then transformed to the specific datasource using the appropriate dialect. You can see how we invoke the ibis-project in the Wren Engine repo branch. It's still a work in progress.

Thus, for databricks support, we'll see if ibis already has integration. If it does, we could easily put that into our roadmap after refactoring to ibis.

[wwwy3y3]

Feel free to upvote here #327 (comment) and share more about your use cases.