Retain precision when casting JSON number to VARCHAR

[findepi]

overview

Normalize JSON numeric values using Java BigDecmial in JSON to VARCHAR cast. Some examples:

1.34e2 -> 13.4
1.34567890123e1 -> 13.4567890123
1.34567890123e8 -> 134567890.123
1.34567890123e11 -> 134567890123
1.34567890123e12-> 1.34567890123E+12

0.000000000000000 -> 0.0
0e1000 -> 0.0
0e-1000 -> 0.0

1 -> 1
100000000000000000000000000000000000000000000000000000000000000000000e-68 -> 1.0
0.100000000000000 -> 0.1

related

• supersedes Fix precision loss when dealing with JSON numbers containing decimal point #28882 (see earlier discussion there)
• fixes Cast from JSON number with decimal point to VARCHAR loses precision #28881

.

• currently based on JsonTypeUtil code cleanup and json parsing test improvements #28915 and Fix numeric precision loss in JSON parsing  #28916

release notes

General
* Improve precision when casting JSON numbers with decimal point to VARCHAR. #28881 

[findepi]

Added Benchmark. Results follow.

Before

Benchmark                                               (jsonType)  (varcharLength)  Mode  Cnt     Score    Error  Units
BenchmarkJsonOperators.benchmarkCastToVarchar         STRING_SHORT            10000  avgt   30    57.470 ±  0.752  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar         STRING_SHORT       2147483647  avgt   30    56.193 ±  1.297  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar        STRING_MEDIUM            10000  avgt   30   176.703 ±  3.310  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar        STRING_MEDIUM       2147483647  avgt   30   174.852 ±  2.693  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar          STRING_LONG            10000  avgt   30  1098.212 ± 20.769  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar          STRING_LONG       2147483647  avgt   30  1101.451 ± 24.898  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar  STRING_WITH_UNICODE            10000  avgt   30    91.809 ±  0.299  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar  STRING_WITH_UNICODE       2147483647  avgt   30    91.903 ±  0.293  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar       NUMBER_INTEGER            10000  avgt   30    62.553 ±  0.780  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar       NUMBER_INTEGER       2147483647  avgt   30    61.504 ±  0.534  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar       NUMBER_DECIMAL            10000  avgt   30   235.296 ±  1.065  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar       NUMBER_DECIMAL       2147483647  avgt   30   236.338 ±  0.889  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar    NUMBER_SCIENTIFIC            10000  avgt   30   178.933 ±  0.826  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar    NUMBER_SCIENTIFIC       2147483647  avgt   30   179.991 ±  1.060  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar         BOOLEAN_TRUE            10000  avgt   30    38.901 ±  0.272  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar         BOOLEAN_TRUE       2147483647  avgt   30    39.677 ±  1.132  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar        BOOLEAN_FALSE            10000  avgt   30    41.334 ±  1.634  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar        BOOLEAN_FALSE       2147483647  avgt   30    39.803 ±  0.112  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar                 NULL            10000  avgt   30    37.529 ±  0.102  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar                 NULL       2147483647  avgt   30    37.812 ±  0.358  ns/op

After

Benchmark                                               (jsonType)  (varcharLength)  Mode  Cnt     Score    Error  Units
BenchmarkJsonOperators.benchmarkCastToVarchar         STRING_SHORT            10000  avgt   30    60.485 ±  4.568  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar         STRING_SHORT       2147483647  avgt   30    55.325 ±  0.272  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar        STRING_MEDIUM            10000  avgt   30   174.716 ±  2.581  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar        STRING_MEDIUM       2147483647  avgt   30   173.241 ±  2.266  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar          STRING_LONG            10000  avgt   30  1115.777 ± 25.443  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar          STRING_LONG       2147483647  avgt   30  1106.917 ± 22.895  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar  STRING_WITH_UNICODE            10000  avgt   30    91.915 ±  0.193  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar  STRING_WITH_UNICODE       2147483647  avgt   30    91.778 ±  0.419  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar       NUMBER_INTEGER            10000  avgt   30    68.107 ±  0.756  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar       NUMBER_INTEGER       2147483647  avgt   30    68.553 ±  0.301  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar       NUMBER_DECIMAL            10000  avgt   30    95.879 ±  0.436  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar       NUMBER_DECIMAL       2147483647  avgt   30    96.344 ±  0.789  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar    NUMBER_SCIENTIFIC            10000  avgt   30    82.827 ±  0.582  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar    NUMBER_SCIENTIFIC       2147483647  avgt   30    82.897 ±  0.924  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar         BOOLEAN_TRUE            10000  avgt   30    38.935 ±  0.156  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar         BOOLEAN_TRUE       2147483647  avgt   30    38.784 ±  0.201  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar        BOOLEAN_FALSE            10000  avgt   30    42.150 ±  0.896  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar        BOOLEAN_FALSE       2147483647  avgt   30    39.383 ±  0.304  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar                 NULL            10000  avgt   30    37.661 ±  0.102  ns/op
BenchmarkJsonOperators.benchmarkCastToVarchar                 NULL       2147483647  avgt   30    37.562 ±  0.187  ns/op

https://jmh.morethan.io/?sources=https://gist.githubusercontent.com/findepi/3eab5435b181fe7df19ffb0a952daa95/raw/7114825186c5a172ef472b9724c87358af7c4873/castToVarchar.01.before.json,https://gist.githubusercontent.com/findepi/3eab5435b181fe7df19ffb0a952daa95/raw/7114825186c5a172ef472b9724c87358af7c4873/castToVarchar.02.after.json

[findepi]

For some reason the benchmark shows performance improvement for the affected case.

I think there results address @dain's concern (#28882 (comment)).

[findepi]

The current implementation suffers from the problem that cast(JSON '...' as ARRAY(VARCHAR)) yields different results than cast(json_parse('...') as ARRAY(VARCHAR)).

[dain]

IMO we should just retain the original text from the JSON. I don't see an upside to normalizing to the text, and the downside is:

1. we loose information this might be meaningful to the user. For example, they might be in an environment where exponentiated numbers are important. Or possible the trailing zeros imply precision of a measurement.
2. performance. Parsing and printing through BigDecimal is not cheap, and object heavy

[findepi]

2. performance. Parsing and printing through BigDecimal is not cheap, and object heavy

Per benchmarks, this turned not to be an issue?
Can you maybe run them too and compare results?

I don't see an upside to normalizing to the text

The "upside" is that cast(JSON '...' as ARRAY(VARCHAR)) and cast(json_parse('...') as ARRAY(VARCHAR)) should yield same results. Can we agree this is expected behavior worth maintaining?

I found a bug in current implementation, which invalidated this assumption. However, combined with #28916 (cherry picked here to run CI), this works as expected, via decimals.

However, with case VALUE_NUMBER_FLOAT -> utf8Slice(parser.getText()) the cast(JSON '...' as ARRAY(VARCHAR)) = cast(json_parse('...') as ARRAY(VARCHAR)) no longer holds.

we loose information this might be meaningful to the user.

We at least agree that lossy cast to varchar is a problem, i.e. #28881 is a bug.
Going through doubles is definitely most lossy from all options considered.