[SPARK-42705][CONNECT] Fix spark.sql to return values from the command #40323
Conversation
|
Is there a similar case on Scala connect client ？ |
I haven't tried Scala client, but yes, it would happen, and this will fix both. |
|
Is there a chance to add a similar case in |
|
@LuciferYang This PR fix it in the connect planner, so should also works for the Scala Client. |
OK, got it |
### What changes were proposed in this pull request?
Fixes `spark.sql` to return values from the command.
### Why are the changes needed?
Currently `spark.sql` doesn't return the result from the commands.
```py
>>> spark.sql("show functions").show()
+--------+
|function|
+--------+
+--------+
```
### Does this PR introduce _any_ user-facing change?
`spark.sql` with commands will return the values.
### How was this patch tested?
Added a test.
Closes #40323 from ueshin/issues/SPARK-42705/sql.
Authored-by: Takuya UESHIN <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 1507a52)
Signed-off-by: Ruifeng Zheng <[email protected]>
|
thank you all, merged into master/branch-3.4 |
| timeZoneId) | ||
| assert(batches.size == 1) | ||
| batches.next() | ||
| assert(batches.hasNext) |
There was a problem hiding this comment.
Sorry for the late reply but how is this code different to the existing one?
val bytes = batches.next()
bytes
Is the same as
batches.next()
The asserts in between don't count as they don't have side effects.
There was a problem hiding this comment.
The batches is an iterator, so batches.size consumes all the data in the iterator to calculate the size.
Then batches.next() would return an empty data? Usually it should throw an Exception, though.
There was a problem hiding this comment.
I see thank you. My understanding with iterators was that non rewindable iterators simply do not have a size method.
But something new learned.
### What changes were proposed in this pull request?
Fixes `spark.sql` to return values from the command.
### Why are the changes needed?
Currently `spark.sql` doesn't return the result from the commands.
```py
>>> spark.sql("show functions").show()
+--------+
|function|
+--------+
+--------+
```
### Does this PR introduce _any_ user-facing change?
`spark.sql` with commands will return the values.
### How was this patch tested?
Added a test.
Closes apache#40323 from ueshin/issues/SPARK-42705/sql.
Authored-by: Takuya UESHIN <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 1507a52)
Signed-off-by: Ruifeng Zheng <[email protected]>
5 participants
What changes were proposed in this pull request?
Fixes
spark.sqlto return values from the command.Why are the changes needed?
Currently
spark.sqldoesn't return the result from the commands.Does this PR introduce any user-facing change?
spark.sqlwith commands will return the values.How was this patch tested?
Added a test.