fix: Disallow lazy backend in read_parquet#2980
Conversation
| elif impl in { | ||
| Implementation.PYSPARK, | ||
| Implementation.DASK, | ||
| Implementation.DUCKDB, | ||
| Implementation.IBIS, | ||
| Implementation.SQLFRAME, | ||
| Implementation.PYSPARK_CONNECT, | ||
| }: | ||
| msg = ( | ||
| f"Expected eager backend, found {impl}.\n\n" | ||
| f"Hint: use nw.scan_csv(source={source}, backend={backend})" | ||
| ) | ||
| raise ValueError(msg) |
There was a problem hiding this comment.
For read_csv I simply wanted to improve on the error message
FBruzzesi
changed the title
read_parquet
| Implementation.DUCKDB, | ||
| Implementation.IBIS, |
There was a problem hiding this comment.
Is it possible these were supposed to be supported for v1 only?
They stand out as the impls that had an interchange support for DataFrame
There was a problem hiding this comment.
Thaaaat's totally possible! However I find it weird that:
read_parquetallows it, butread_csvdoesn'tfrom_nativein the return statement has theeager_only=Trueflag, not theeager_or_interchange_only
🤔
There was a problem hiding this comment.
yeah true, I guess you could investigate 🧐 the blame?
There was a problem hiding this comment.
#1725 introduced full support for duckdb, and added duckdb in read_parquet
#2000 introduced full support for ibis, and added ibis in read_parquet
These are way after interchange protocol support, so it seems completely accidental, and we just missed it. Both PRs were 2k+ lines changes and we had no test to prevent that
|
nice one thanks both! |
3 participants
What type of PR is this? (check all applicable)
Related issues
read_parquetallow lazy backend #2979Checklist
If you have comments or can explain your changes, please do so below