[SPARK-1608] [SQL] Fix Cast.nullable when cast from StringType to NumericType/TimestampType. #532
Conversation
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Hey, thanks for this fix too! I'm a little torn about adding the nullability, foldability and datatype checks to all the evaluation tests. At the very least, we need to use named parameters with Boolean arguments to keep things readable. However, I think a better option might be to separate these checks out into their own unit tests. Doing so might decease the amount of redundancy and make it more obvious what cases are actually being tested. Mostly I want to keep a very concise way to write expression evaluation unit tests so that it is easy to keep adding them. What do you think? |
|
Hi, thank you for your reply. I thought that all expressions should be checked in the same way but now I agree that the tests are not readable and concise. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Thanks. Merged. |
…ericType/TimestampType. `Cast.nullable` should be `true` when cast from `StringType` to `NumericType` or `TimestampType`. Because if `StringType` expression has an illegal number string or illegal timestamp string, the casted value becomes `null`. Author: Takuya UESHIN <[email protected]> Closes #532 from ueshin/issues/SPARK-1608 and squashes the following commits: 065d37c [Takuya UESHIN] Add tests to check nullabilities of cast expressions. f278ed7 [Takuya UESHIN] Revert test to keep it readable and concise. 9fc9380 [Takuya UESHIN] Fix Cast.nullable when cast from StringType to NumericType/TimestampType. (cherry picked from commit 8e37ed6) Signed-off-by: Reynold Xin <[email protected]>
…ericType/TimestampType. `Cast.nullable` should be `true` when cast from `StringType` to `NumericType` or `TimestampType`. Because if `StringType` expression has an illegal number string or illegal timestamp string, the casted value becomes `null`. Author: Takuya UESHIN <[email protected]> Closes apache#532 from ueshin/issues/SPARK-1608 and squashes the following commits: 065d37c [Takuya UESHIN] Add tests to check nullabilities of cast expressions. f278ed7 [Takuya UESHIN] Revert test to keep it readable and concise. 9fc9380 [Takuya UESHIN] Fix Cast.nullable when cast from StringType to NumericType/TimestampType.
apache#532) * [SPARK-25299] Use the shuffle writer plugin for the SortShuffleWriter. * Remove unused * Handle empty partitions properly. * Adjust formatting * Don't close streams twice. Because compressed output streams don't like it. * Clarify comment
apache#532) * [SPARK-25299] Use the shuffle writer plugin for the SortShuffleWriter. * Remove unused * Handle empty partitions properly. * Adjust formatting * Don't close streams twice. Because compressed output streams don't like it. * Clarify comment
4 participants
Cast.nullableshould betruewhen cast fromStringTypetoNumericTypeorTimestampType.Because if
StringTypeexpression has an illegal number string or illegal timestamp string, the casted value becomesnull.