Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Fix partitioning SQL scans on empty tables #2885

Merged

Conversation

desmondcheongzx
Copy link
Contributor

@desmondcheongzx desmondcheongzx commented Sep 23, 2024

When scanning an empty SQL table, if a user specifies num_partitions > 1, then we get TypeError: unsupported operand type(s) for -: 'NoneType' and 'NoneType' or Failed to get partition bounds: {self._partition_col} is not a numeric or temporal type, and cannot be used for partitioning.

These are both results of attempts to partition the scan using a column with no data. Despite the table not having rows, we attempt to fulfill the user's request to use num_partitions > 2, only to run into errors because there is no min and max value available compute partition range sizes on.

Furthermore, in some cases SQL databases do not return type information when a table has no rows, so an empty integer column might be read as an empty string column which cannot be used for partitioning.

We fix this by capping the number of scan tasks by the number of rows in the scan table. If there are no rows, we don't attempt partitioning, and we simply generate 0 scan tasks.

@github-actions github-actions bot added the bug Something isn't working label Sep 23, 2024
Copy link

codspeed-hq bot commented Sep 23, 2024

CodSpeed Performance Report

Merging #2885 will not alter performance

Comparing desmondcheongzx:fix-sql-scan-empty (ed957b7) with main (b0f31e3)

Summary

✅ 17 untouched benchmarks

@desmondcheongzx desmondcheongzx changed the title [BUG] Fix SQL scans on empty tables [BUG] Fix partitioning SQL scans on empty tables Sep 23, 2024
Copy link
Contributor

@colin-ho colin-ho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this on @desmondcheongzx !!

@desmondcheongzx desmondcheongzx merged commit 29be743 into Eventual-Inc:main Sep 23, 2024
44 checks passed
@desmondcheongzx desmondcheongzx deleted the fix-sql-scan-empty branch September 23, 2024 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants