Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Avoid reconstructing sql query in read_sql #2818

Merged
merged 2 commits into from
Sep 17, 2024

Conversation

colin-ho
Copy link
Contributor

@colin-ho colin-ho commented Sep 9, 2024

When Daft executes a read_sql scan task, it calls the def read_sql function in table_io.py. This function then calls the .read method on the SQLConnection object. However, the .read method will reconstruct the sql query and add another layer of subqueries, which is unnecessary.

This is because the .read method constructs a sql query given additional predicates, projections, and limits, then executes it. However, the scan task is already given a constructed query with pushdowns applied, so this reconstruction is unnecessary.

This PR removes the .read method and instead exposes the execute_sql_query method. Having .read do construction and execution together is confusing.

@github-actions github-actions bot added the bug Something isn't working label Sep 9, 2024
Copy link

codspeed-hq bot commented Sep 9, 2024

CodSpeed Performance Report

Merging #2818 will degrade performances by 83.68%

Comparing colin/unncessary-sql-subquery (ae62105) with main (d30e62a)

Summary

❌ 1 regressions
✅ 15 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark main colin/unncessary-sql-subquery Change
test_show[100 Small Files] 48.7 ms 298.6 ms -83.68%

@jaychia jaychia self-requested a review September 17, 2024 00:43
@colin-ho colin-ho merged commit 72b1440 into main Sep 17, 2024
38 of 39 checks passed
@colin-ho colin-ho deleted the colin/unncessary-sql-subquery branch September 17, 2024 00:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants