From e996db40200d3b640cc139ddc175d303e001c6c1 Mon Sep 17 00:00:00 2001 From: Max Gekk Date: Fri, 25 Dec 2020 09:59:07 +0300 Subject: [PATCH 1/2] Update SQL migration guide --- docs/sql-migration-guide.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md index 4b6c2266387f5..b49e4de120ea2 100644 --- a/docs/sql-migration-guide.md +++ b/docs/sql-migration-guide.md @@ -28,6 +28,11 @@ license: | - In Spark 3.2, the meta-characters `\n` and `\t` are escaped in the `show()` action. In Spark 3.1 or earlier, the two metacharacters are output as it is. + - In Spark 3.2, `AnalysisException` is replaced by its sub-classes that thrown for tables from Hive external catalog in the following situations: + * `ALTER TABLE .. ADD PARTITION` throws `PartitionsAlreadyExistException` if new partition exists already + * `ALTER TABLE .. DROP PARTITION` throws `NoSuchPartitionsException` for not existing partitions + * `ALTER TABLE .. RENAME TO PARTITION` throws `PartitionAlreadyExistsException` when the target partition already exists + ## Upgrading from Spark SQL 3.0 to 3.1 - In Spark 3.1, statistical aggregation function includes `std`, `stddev`, `stddev_samp`, `variance`, `var_samp`, `skewness`, `kurtosis`, `covar_samp`, `corr` will return `NULL` instead of `Double.NaN` when `DivideByZero` occurs during expression evaluation, for example, when `stddev_samp` applied on a single element set. In Spark version 3.0 and earlier, it will return `Double.NaN` in such case. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.statisticalAggregate` to `true`. From cb78094f9ac74ff7021f1aedb99a04155feb8714 Mon Sep 17 00:00:00 2001 From: Max Gekk Date: Fri, 25 Dec 2020 10:17:29 +0300 Subject: [PATCH 2/2] Update for other versions --- docs/sql-migration-guide.md | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md index b49e4de120ea2..cbb1de53c8896 100644 --- a/docs/sql-migration-guide.md +++ b/docs/sql-migration-guide.md @@ -28,10 +28,7 @@ license: | - In Spark 3.2, the meta-characters `\n` and `\t` are escaped in the `show()` action. In Spark 3.1 or earlier, the two metacharacters are output as it is. - - In Spark 3.2, `AnalysisException` is replaced by its sub-classes that thrown for tables from Hive external catalog in the following situations: - * `ALTER TABLE .. ADD PARTITION` throws `PartitionsAlreadyExistException` if new partition exists already - * `ALTER TABLE .. DROP PARTITION` throws `NoSuchPartitionsException` for not existing partitions - * `ALTER TABLE .. RENAME TO PARTITION` throws `PartitionAlreadyExistsException` when the target partition already exists + - In Spark 3.2, `ALTER TABLE .. RENAME TO PARTITION` throws `PartitionAlreadyExistsException` instead of `AnalysisException` for tables from Hive external when the target partition already exists. ## Upgrading from Spark SQL 3.0 to 3.1 @@ -69,6 +66,16 @@ license: | - Since Spark 3.1, CHAR/CHARACTER and VARCHAR types are supported in the table schema. Table scan/insertion will respect the char/varchar semantic. If char/varchar is used in places other than table schema, an exception will be thrown (CAST is an exception that simply treats char/varchar as string like before). To restore the behavior before Spark 3.1, which treats them as STRING types and ignores a length parameter, e.g. `CHAR(4)`, you can set `spark.sql.legacy.charVarcharAsString` to `true`. + - In Spark 3.1, `AnalysisException` is replaced by its sub-classes that are thrown for tables from Hive external catalog in the following situations: + * `ALTER TABLE .. ADD PARTITION` throws `PartitionsAlreadyExistException` if new partition exists already + * `ALTER TABLE .. DROP PARTITION` throws `NoSuchPartitionsException` for not existing partitions + +## Upgrading from Spark SQL 3.0.1 to 3.0.2 + + - In Spark 3.0.2, `AnalysisException` is replaced by its sub-classes that are thrown for tables from Hive external catalog in the following situations: + * `ALTER TABLE .. ADD PARTITION` throws `PartitionsAlreadyExistException` if new partition exists already + * `ALTER TABLE .. DROP PARTITION` throws `NoSuchPartitionsException` for not existing partitions + ## Upgrading from Spark SQL 3.0 to 3.0.1 - In Spark 3.0, JSON datasource and JSON function `schema_of_json` infer TimestampType from string values if they match to the pattern defined by the JSON option `timestampFormat`. Since version 3.0.1, the timestamp type inference is disabled by default. Set the JSON option `inferTimestamp` to `true` to enable such type inference. @@ -262,6 +269,12 @@ license: | * The decimal string representation can be different between Hive 1.2 and Hive 2.3 when using `TRANSFORM` operator in SQL for script transformation, which depends on hive's behavior. In Hive 1.2, the string representation omits trailing zeroes. But in Hive 2.3, it is always padded to 18 digits with trailing zeroes if necessary. +## Upgrading from Spark SQL 2.4.7 to 2.4.8 + + - In Spark 2.4.8, `AnalysisException` is replaced by its sub-classes that are thrown for tables from Hive external catalog in the following situations: + * `ALTER TABLE .. ADD PARTITION` throws `PartitionsAlreadyExistException` if new partition exists already + * `ALTER TABLE .. DROP PARTITION` throws `NoSuchPartitionsException` for not existing partitions + ## Upgrading from Spark SQL 2.4.5 to 2.4.6 - In Spark 2.4.6, the `RESET` command does not reset the static SQL configuration values to the default. It only clears the runtime SQL configuration values.