[SPARK-27990][SPARK-29903][PYTHON] Add recursiveFileLookup option to Python DataFrameReader #26718
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #114631 has finished for PR 26718 at commit
|
nchammas
commented
Nov 29, 2019
|
|
||
| @since(1.4) | ||
| def parquet(self, *paths): | ||
| def parquet(self, *paths, **options): |
Contributor
Author
There was a problem hiding this comment.
To support Python 2, we need to say **options instead of recursiveFileLookup=None because Python 2 doesn't support keyword-only arguments.
|
Test build #114638 has finished for PR 26718 at commit
|
Contributor
Author
|
cc @gatorsmile and @cloud-fan, per the discussion here. For some reason, I can't add you as reviewers using the GitHub interface. |
HyukjinKwon
changed the title
HyukjinKwon
reviewed
Nov 30, 2019
Member
HyukjinKwon
left a comment
HyukjinKwon
left a comment
There was a problem hiding this comment.
@nchammas, can we at least update/test in streaming.py too?
Contributor
Author
|
Certainly! Oversight on my part. |
|
Test build #114673 has finished for PR 26718 at commit
|
HyukjinKwon
reviewed
Dec 1, 2019
Member
HyukjinKwon
left a comment
HyukjinKwon
left a comment
There was a problem hiding this comment.
Haven't closely checked but seems fine. If this doesn't get merged, I will take a look again and get this in.
Member
|
Merged to master. |
attilapiros
pushed a commit
to attilapiros/spark
that referenced
this pull request
Dec 6, 2019
…Python DataFrameReader ### What changes were proposed in this pull request? As a follow-up to apache#24830, this PR adds the `recursiveFileLookup` option to the Python DataFrameReader API. ### Why are the changes needed? This PR maintains Python feature parity with Scala. ### Does this PR introduce any user-facing change? Yes. Before this PR, you'd only be able to use this option as follows: ```python spark.read.option("recursiveFileLookup", True).text("test-data").show() ``` With this PR, you can reference the option from within the format-specific method: ```python spark.read.text("test-data", recursiveFileLookup=True).show() ``` This option now also shows up in the Python API docs. ### How was this patch tested? I tested this manually by creating the following directories with dummy data: ``` test-data ├── 1.txt └── nested └── 2.txt test-parquet ├── nested │ ├── _SUCCESS │ ├── part-00000-...-.parquet ├── _SUCCESS ├── part-00000-...-.parquet ``` I then ran the following tests and confirmed the output looked good: ```python spark.read.parquet("test-parquet", recursiveFileLookup=True).show() spark.read.text("test-data", recursiveFileLookup=True).show() spark.read.csv("test-data", recursiveFileLookup=True).show() ``` `python/pyspark/sql/tests/test_readwriter.py` seems pretty sparse. I'm happy to add my tests there, though it seems we have been deferring testing like this to the Scala side of things. Closes apache#26718 from nchammas/SPARK-27990-recursiveFileLookup-python. Authored-by: Nicholas Chammas <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
Closed
47 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
As a follow-up to #24830, this PR adds the
recursiveFileLookupoption to the Python DataFrameReader API.Why are the changes needed?
This PR maintains Python feature parity with Scala.
Does this PR introduce any user-facing change?
Yes.
Before this PR, you'd only be able to use this option as follows:
With this PR, you can reference the option from within the format-specific method:
This option now also shows up in the Python API docs.
How was this patch tested?
I tested this manually by creating the following directories with dummy data:
I then ran the following tests and confirmed the output looked good:
python/pyspark/sql/tests/test_readwriter.pyseems pretty sparse. I'm happy to add my tests there, though it seems we have been deferring testing like this to the Scala side of things.