Skip to content

[SPARK-21335] [DOC] doc changes for disallowed un-aliased subquery use case #21647

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cnZach wants to merge 2 commits into from
Closed

[SPARK-21335] [DOC] doc changes for disallowed un-aliased subquery use case #21647

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c611a11
add documentaion for disallowed un-aliased subquery use case
cnZach Jun 27, 2018
bebc3a8
update the use case description
cnZach Jun 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/sql-programming-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2017,6 +2017,7 @@ working with timestamps in `pandas_udf`s to get the best performance, see
- Literal values used in SQL operations are converted to DECIMAL with the exact precision and scale needed by them.
- The configuration `spark.sql.decimalOperations.allowPrecisionLoss` has been introduced. It defaults to `true`, which means the new behavior described here; if set to `false`, Spark uses previous rules, ie. it doesn't adjust the needed scale to represent the values and it returns NULL if an exact representation of the value is not possible.
- In PySpark, `df.replace` does not allow to omit `value` when `to_replace` is not a dictionary. Previously, `value` could be omitted in the other cases and had `None` by default, which is counterintuitive and error-prone.
- Un-aliased subquery is supported by Spark SQL for a long time. Its semantic was not well defined and had confusing behaviors. Since Spark 2.3, we invalid a weird use case: `SELECT v.i from (SELECT i FROM v)`. Now this query will throw analysis exception because users should not be able to use the qualifier inside a subquery. See [SPARK-20690](https://issues.apache.org/jira/browse/SPARK-20690) and [SPARK-21335](https://issues.apache.org/jira/browse/SPARK-21335) for details.
Copy link
Member

@HyukjinKwon HyukjinKwon Jun 27, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a big deal but please consider:

Un-aliased subquery is supported by Spark SQL for a long time. Its semantic was not well defined and had confusing behaviors. Since Spark 2.3, we invalid a weird use case: SELECT v.i from (SELECT i FROM v)

->

Un-aliased subquery's semantic has not been well defined with confusing behaviors. Since Spark 2.3, we invalidate such confusing cases, for example, SELECT v.i from (SELECT i FROM v).

Copy link
Member

@HyukjinKwon HyukjinKwon Jun 27, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also consider:

Now this query will throw analysis exception because users should not be able to use the qualifier inside a subquery.

->

The cases throw an analysis exception now because users should not be able to use the qualifier inside a subquery.

Copy link
Member

@HyukjinKwon HyukjinKwon Jun 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for details. -> for more details.


## Upgrading From Spark SQL 2.1 to 2.2

Expand Down