Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] Initializes daft-sql and defines the daft.sql(..) function. #2559

Merged
merged 7 commits into from
Jul 25, 2024

Conversation

RCHowell
Copy link
Contributor

@RCHowell RCHowell commented Jul 24, 2024

This pull request adds the daft-sql module to the Rust side, and defines a daft.sql(..) method along with a placeholder test.

Then intention of this PR is to establish the structure and plumbing before introducing any functionality.

@RCHowell RCHowell changed the title Initializes daft-sql and defines the daft.sql(..) function. [FEAT] Initializes daft-sql and defines the daft.sql(..) function. Jul 24, 2024
@github-actions github-actions bot added the enhancement New feature or request label Jul 24, 2024
@RCHowell RCHowell mentioned this pull request Jul 24, 2024
5 tasks
@RCHowell
Copy link
Contributor Author

Demo

demo

import daft
from daft.sql.sql import SQLCatalog

# register dataframes to the catalog
catalog = SQLCatalog({
  "example": daft.from_pydict({ "a": [1, 2, 3] }),

})

# invoke daft.sql()
df = daft.sql("SELECT * FROM example", catalog)
df.collect()

Copy link
Collaborator

@universalmind303 universalmind303 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so i think the work that I did in #2558 will compliment this pretty well.

#2558 adds a lot of sql parsing logic, and this PR sets up a lot of the python frontend that #2558 is missing.

@RCHowell
Copy link
Contributor Author

RCHowell commented Jul 25, 2024

Nice! I saw that we started to overlap on the Rust side — you were able to get a lot done so quickly. If you want to focus on the Rust side, I will rebase from #2558 when I have bandwidth. Otherwise, you are welcome to take the Python frontend things you wish and change up the names like SQLCatalog->SQLContext.

@universalmind303
Copy link
Collaborator

Nice! I saw that we started to overlap on the Rust side — you were able to get a lot done so quickly. If you want to focus on the Rust side, I will rebase from #2558 when I have bandwidth.

@RCHowell no need, we can merge this in first and I can rebase off of yours.

import pytest

import daft
from daft.sql.sql import SQLCatalog
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: is there any way we can have this just from daft.sql instead of `from daft.sql.sql

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I totally agree, but struggled without moving existing things. I had a circular import error by adding things in sql.py to the sql/__init__.py because of the existing files.

Is moving sql_scan and sql_connection to avoid the circular dependency a viable option?

daft/sql/sql.py Outdated Show resolved Hide resolved
daft/io/_sql.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@universalmind303 universalmind303 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few small nits, but overall looks good! great work @RCHowell!

@universalmind303 universalmind303 enabled auto-merge (squash) July 25, 2024 20:57
@universalmind303 universalmind303 merged commit c329ae9 into Eventual-Inc:main Jul 25, 2024
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants