feat: add dry run to the read_gbq function #979
Conversation
product-auto-label
bot
added
size: s
shuoweil
added
the
kokoro:force-run
yoshi-kokoro
removed
the
kokoro:force-run
|
@antoineeripret Could you please check the failed tests? Thanks a lot. |
product-auto-label
bot
added
size: m
|
@shuoweil , I've added a new commit with some changes to fix tests. I've ran |
Hi @antoineeripret, could you please check the failed check please? It should be a quick fix. Thanks a lot. |
|
Hi @shuoweil, the last commit should fix it. Got the following on my local env: python -m black --check docs pandas_gbq tests noxfile.py setup.py
All done! ✨ 🍰 ✨
45 files would be left unchanged. |
pandas_gbq/gbq_connector.py
Outdated
| # we need to get it from the query result | ||
| # For query_and_wait_via_client_library, the RowIterator should have job set | ||
| raise ValueError("Cannot access QueryJob from RowIterator for dry_run") | ||
| return query_job.total_bytes_processed / 1024**3 |
There was a problem hiding this comment.
Could we simply return query_job.total_bytes_processed without further processing?
Reasons:
- The
total_bytes_processedhas integer type, which is more precise than a float type - For small tables (ones with 1-10 MB sizes), converting the size to GB makes the result less readable
- It aligns more with the behavior of BigQuery Python client to return size in bytes.
Generally speaking, we want the caller of this function to perform unit conversions.
There was a problem hiding this comment.
@sycai, good call ! I've though about my own usage, but didn't think about the bigger picture here. I'll commit the change. :)
sycai
left a comment
sycai
left a comment
There was a problem hiding this comment.
Thank you! I think we should be good to go once the doc and tests are updated.
|
@sycai : updated :) |
|
Looks like there's a lint error. Could you fix it? Thanks a lot! |
| ------- | ||
| df: DataFrame | ||
| DataFrame representing results of query. | ||
| df: DataFrame or float |
| DataFrame representing results of query. | ||
| df: DataFrame or float | ||
| DataFrame representing results of query. If ``dry_run=True``, returns | ||
| a float representing the amount of data that would be processed (in bytes). |
There was a problem hiding this comment.
doc nit: "returns an int representing ..."
|
@antoineeripret I believe lint fails. Could you please update it? It still fails with the new commit. |
5 participants
This change allows the user to run a dry run query using the
read_gbqfunction. Instead of returning apd.DataFrame, the behavior is changed and the amount of data processed (in GB) is returned.