Skip to content

Commit f37ab0d

Browse files
committed
Add note about spark.executor.heartbeatInterval change to migration guide
1 parent 777b450 commit f37ab0d

File tree

1 file changed

+8
-0
lines changed

1 file changed

+8
-0
lines changed

docs/sql-migration-guide-upgrade.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,14 @@ license: |
124124

125125
- In Spark version 2.4, when a spark session is created via `cloneSession()`, the newly created spark session inherits its configuration from its parent `SparkContext` even though the same configuration may exist with a different value in its parent spark session. Since Spark 3.0, the configurations of a parent `SparkSession` have a higher precedence over the parent `SparkContext`.
126126

127+
## Upgrading from Spark SQL 2.4 to 2.4.1
128+
129+
- The value of `spark.executor.heartbeatInterval`, when specified without units like "30" rather than "30s", was
130+
inconsistently interpreted as both seconds and milliseconds in Spark 2.4.0 in different parts of the code.
131+
Unitless values are now consistently interpreted as milliseconds. Applications that set values like "30"
132+
need to specify a value with units like "30s" now, to avoid being interpreted as milliseconds; otherwise,
133+
the extremely short interval that results will likely cause applications to fail.
134+
127135
## Upgrading From Spark SQL 2.3 to 2.4
128136

129137
- In Spark version 2.3 and earlier, the second parameter to array_contains function is implicitly promoted to the element type of first array type parameter. This type promotion can be lossy and may cause `array_contains` function to return wrong result. This problem has been addressed in 2.4 by employing a safer type promotion mechanism. This can cause some change in behavior and are illustrated in the table below.

0 commit comments

Comments
 (0)