[SPARK-41036][CONNECT][PYTHON] columns API should use schema API to avoid data fetching
#38546
Conversation
|
Can one of the admins verify this patch? |
There was a problem hiding this comment.
maybe we don't need to cache columns, just return self.schema.names?
There was a problem hiding this comment.
I would prefer to not depend on the underly API when doing caching...
E.g. what if someday the cache on the schema is gone but this API is not aware of it, etc.
Basically do not make assumptions :)
There was a problem hiding this comment.
Hmmm .. why do we need _cache? I think we can just remove.
There was a problem hiding this comment.
hmm if users call this API multiple times, we only need one gRPC. This should be useful right?
something like:
df.columns()
xxxx
xxx
df.columns()
xxxx
xxxx
df.columns()
There was a problem hiding this comment.
if you think this is a bit over-engineering I can remove.
There was a problem hiding this comment.
For that case, we should probably have a proper cache layer instead of doing this alone in names.
There was a problem hiding this comment.
we can do that for all cases for metadata-ish cases. e.g., schema or even collected results
There was a problem hiding this comment.
I removed this caching stuff in this PR. After we pretty much support enough API, we can go back to build a cache layer for all metadata like API to save RPC calls.
…to avoid data fetching.
amaliujia
force-pushed
the
improve_python_columns
branch
from
be1216f to
c6ae61b
Compare
|
merged into master |
…o avoid data fetching ### What changes were proposed in this pull request? Current `columns` is implemented based on `limit` which runs a job to fetch data and get schema from the data collection. However a more efficient way is to call `schema` API which only need to analyze the plan without collect data. This approach should be more efficient in most of the cases. ### Why are the changes needed? Efficiency ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? UT Closes apache#38546 from amaliujia/improve_python_columns. Authored-by: Rui Wang <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>
4 participants
What changes were proposed in this pull request?
Current
columnsis implemented based onlimitwhich runs a job to fetch data and get schema from the data collection. However a more efficient way is to callschemaAPI which only need to analyze the plan without collect data. This approach should be more efficient in most of the cases.Why are the changes needed?
Efficiency
Does this PR introduce any user-facing change?
NO
How was this patch tested?
UT