Skip to content

[SPARK-28023][SQL] Add trim logic in UTF8String's toInt/toLong to make it consistent with other string-numeric casting #26622

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yaooqinn wants to merge 16 commits into from
Closed

[SPARK-28023][SQL] Add trim logic in UTF8String's toInt/toLong to make it consistent with other string-numeric casting #26622

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
48d5a9f
[WIP][SPARK-28023][Test] Cheap UTF8String Trim
yaooqinn Nov 21, 2019
05ab098
benchmark without trim
yaooqinn Nov 21, 2019
c769c40
benchmark without trim
yaooqinn Nov 21, 2019
ee94f98
benchmark with copy trim
yaooqinn Nov 21, 2019
4fa535b
benchmark with non copy trim
yaooqinn Nov 21, 2019
2261433
benchmark with non copy trim r2
yaooqinn Nov 21, 2019
cce5a55
benchmark with copy trim r2
yaooqinn Nov 21, 2019
4fd810a
inside trim
yaooqinn Nov 21, 2019
cf2c243
rm left spaces
yaooqinn Nov 21, 2019
98892fe
trim v.s inside trim
yaooqinn Nov 21, 2019
d0bade1
method local
yaooqinn Nov 21, 2019
5b75d37
benchmark
yaooqinn Nov 21, 2019
09f38f7
fix bug and benchmark
yaooqinn Nov 21, 2019
76639b7
clean benchmark
yaooqinn Nov 21, 2019
f37f467
control chars
yaooqinn Nov 22, 2019
d5c2a40
m guide
yaooqinn Nov 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -484,7 +484,7 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
private[this] def castToLong(from: DataType): Any => Any = from match {
case StringType =>
val result = new LongWrapper()
buildCast[UTF8String](_, s => if (s.toLong(result)) result.value else null)
buildCast[UTF8String](_, s => if (s.trim().toLong(result)) result.value else null)
case BooleanType =>
buildCast[Boolean](_, b => if (b) 1L else 0L)
case DateType =>
Expand All @@ -501,7 +501,7 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
private[this] def castToInt(from: DataType): Any => Any = from match {
case StringType =>
val result = new IntWrapper()
buildCast[UTF8String](_, s => if (s.toInt(result)) result.value else null)
buildCast[UTF8String](_, s => if (s.trim().toInt(result)) result.value else null)
case BooleanType =>
buildCast[Boolean](_, b => if (b) 1 else 0)
case DateType =>
Expand Down Expand Up @@ -1418,7 +1418,7 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
(c, evPrim, evNull) =>
code"""
UTF8String.IntWrapper $wrapper = new UTF8String.IntWrapper();
if ($c.toInt($wrapper)) {
if ($c.trim().toInt($wrapper)) {
$evPrim = $wrapper.value;
} else {
$evNull = true;
Expand Down Expand Up @@ -1447,7 +1447,7 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
(c, evPrim, evNull) =>
code"""
UTF8String.LongWrapper $wrapper = new UTF8String.LongWrapper();
if ($c.toLong($wrapper)) {
if ($c.trim().toLong($wrapper)) {
$evPrim = $wrapper.value;
} else {
$evNull = true;
Expand Down
4 changes: 2 additions & 2 deletions sql/core/benchmarks/CastBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Java HotSpot(TM) 64-Bit Server VM 1.8.0_231-b11 on Mac OS X 10.15.1
Intel(R) Core(TM) i5-5287U CPU @ 2.90GHz
Benchmark trim the string: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
cast(str as int) as c_int 2478 3669 1046 1.7 604.9 1.0X
cast(str as long) as c_long 1439 1548 94 2.8 351.4 1.7X
cast(str as int) as c_int 3169 3530 610 1.3 773.6 1.0X
cast(str as long) as c_long 1812 1881 60 2.3 442.4 1.7X
Copy link
Member Author

@yaooqinn yaooqinn Nov 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cloud-fan this is master branch to original trim result.