Fix wrong precision by tangjiangling · Pull Request #808 · ClickHouse/clickhouse-java

tangjiangling · 2022-01-16T16:20:02Z

Fixes: #807

com.clickhouse.jdbc.ClickHouseDatabaseMetaData#getColumns
returns the wrong precision, which is against the JDBC standard.

CLAassistant · 2022-01-16T16:20:07Z

All committers have signed the CLA.

tangjiangling · 2022-01-16T16:21:13Z

A related issue(trinodb/trino#10541)

tangjiangling · 2022-01-16T16:22:35Z

cc: @ebyhr @hashhar

github-actions · 2022-01-16T16:36:22Z

Benchmark                                 (client)  (connection)  (statement)  (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc1         reuse       normal  object  thrpt   20   240.527 ±  33.461  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1         reuse     prepared  object  thrpt   20   237.337 ±  26.340  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1           new       normal  object  thrpt   20   235.198 ±  27.903  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1           new     prepared  object  thrpt   20   234.410 ±  22.012  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc         reuse       normal  object  thrpt   20   247.870 ±  27.479  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc         reuse     prepared  object  thrpt   20   248.048 ±  26.457  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc           new       normal  object  thrpt   20   246.023 ±  29.453  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc           new     prepared  object  thrpt   20   241.346 ±  25.222  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1         reuse       normal  object  thrpt   20  1576.755 ± 131.857  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1         reuse     prepared  object  thrpt   20  1646.127 ± 108.495  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1           new       normal  object  thrpt   20  1658.941 ±  70.198  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1           new     prepared  object  thrpt   20  1687.920 ±  64.342  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc         reuse       normal  object  thrpt   20  1040.702 ±  92.676  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc         reuse     prepared  object  thrpt   20  1042.248 ±  97.646  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc           new       normal  object  thrpt   20  1051.719 ±  93.958  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc           new     prepared  object  thrpt   20  1006.116 ±  90.380  ops/s

tangjiangling · 2022-01-16T17:39:44Z

The second commit describes:

In method com.clickhouse.jdbc.ClickHouseDatabaseMetaData#getColumns,
we should map DECIMAL_DIGITS and CHAR_OCTET_LENGTH to int32(0),
otherwise it will lead to incorrect results.

Why?

The original implementation maps DECIMAL_DIGITS and CHAR_OCTET_LENGTH to int32(null), which causes the corresponding field type to become com.clickhouse.client.data.ClickHouseEmptyValue, which is unable to update the new value, so it will end up with the wrong result.

github-actions · 2022-01-16T17:46:34Z

Benchmark                                 (client)  (connection)  (statement)  (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc1         reuse       normal  object  thrpt   20   224.853 ±  29.809  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1         reuse     prepared  object  thrpt   20   214.132 ±  25.693  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1           new       normal  object  thrpt   20   217.588 ±  26.057  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1           new     prepared  object  thrpt   20   214.580 ±  27.600  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc         reuse       normal  object  thrpt   20   228.626 ±  25.056  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc         reuse     prepared  object  thrpt   20   231.906 ±  28.852  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc           new       normal  object  thrpt   20   231.754 ±  27.423  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc           new     prepared  object  thrpt   20   226.982 ±  28.056  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1         reuse       normal  object  thrpt   20  1236.902 ±  87.694  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1         reuse     prepared  object  thrpt   20  1226.242 ± 132.556  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1           new       normal  object  thrpt   20  1214.646 ±  77.618  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1           new     prepared  object  thrpt   20  1233.896 ±  87.188  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc         reuse       normal  object  thrpt   20   786.241 ±  78.995  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc         reuse     prepared  object  thrpt   20   784.618 ±  83.369  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc           new       normal  object  thrpt   20   757.054 ±  83.106  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc           new     prepared  object  thrpt   20   842.684 ±  59.096  ops/s

zhicwu · 2022-01-16T23:33:44Z

Thanks @tangjiangling for the pull request :) I don't mind to create another patch release, but I'll need to run integration test against trino to see if there's more issues.

In the mean time, before merging the PR, would you mind to sign CLA and rebase? I changed merge target from master to develop, and I'll add some more tests.

tangjiangling · 2022-01-17T03:35:32Z

would you mind to sign CLA and rebase?

signed.

What does rebase mean?

Is it a combination of all the "Merge pull request ClickHouse#..." into a single commit?

tangjiangling · 2022-01-17T03:57:43Z

would you mind to sign CLA and rebase?

Sorry, I forgot to set the email, I will do that soon.

tangjiangling · 2022-01-17T04:09:57Z

Sorry, I forgot to set the email, I will do that soon.

done.

github-actions · 2022-01-17T04:26:01Z

Benchmark                                 (client)  (connection)  (statement)  (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc1         reuse       normal  object  thrpt   20   239.145 ±  34.485  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1         reuse     prepared  object  thrpt   20   232.703 ±  26.263  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1           new       normal  object  thrpt   20   231.571 ±  25.240  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1           new     prepared  object  thrpt   20   232.121 ±  27.181  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc         reuse       normal  object  thrpt   20   242.690 ±  29.855  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc         reuse     prepared  object  thrpt   20   240.499 ±  26.050  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc           new       normal  object  thrpt   20   242.497 ±  28.161  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc           new     prepared  object  thrpt   20   242.667 ±  27.760  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1         reuse       normal  object  thrpt   20  1579.735 ± 105.663  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1         reuse     prepared  object  thrpt   20  1611.489 ±  73.549  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1           new       normal  object  thrpt   20  1594.689 ±  85.338  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1           new     prepared  object  thrpt   20  1601.815 ±  95.206  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc         reuse       normal  object  thrpt   20  1016.533 ±  87.853  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc         reuse     prepared  object  thrpt   20   998.676 ±  82.197  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc           new       normal  object  thrpt   20   940.991 ± 116.601  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc           new     prepared  object  thrpt   20   999.509 ±  87.986  ops/s

zhicwu · 2022-01-17T05:05:58Z

Is it a combination of all the "Merge pull request ClickHouse#..." into a single commit?

Yes, squash may work too.

incorrect precision

I think we should consider both cases. When the column is numeric, we use column.getPrecision(), otherwise fall back to column.getDataType().getByteLength(). Make sense?

changed toInt32(null) to toInt32(0)

I'll need to look into this tonight and add some more tests accordingly.

Update:
DECIMAL_DIGITS is nullable according to javadoc, and CHAR_OCTET_LENGTH is nullable as well according to other JDBC driver implementation. The problem here is that toInt32(null) resulted in a column with type Nullable(Nothing), which was mapped to ClickHouseEmptyValue. I think it's better to change it to cast(null as Nullable(Int32)).

I'll fix the these issues and release 0.3.2-patch3 after testing against trino master branch.

tangjiangling · 2022-01-17T06:14:49Z

I think we should consider both cases. When the column is numeric, we use column.getPrecision(), otherwise fall back to column.getDataType().getByteLength(). Make sense?

+1, I will update the current PR later this evening.

tangjiangling · 2022-01-23T21:35:31Z

+1, I will update the current PR later this evening.

I fixed it on the basis of this #815 .

github-actions · 2022-01-23T21:43:29Z

Benchmark                                 (client)  (connection)  (statement)  (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc1         reuse       normal  object  thrpt   20   208.988 ±  28.144  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1         reuse     prepared  object  thrpt   20   206.678 ±  26.898  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1           new       normal  object  thrpt   20   203.693 ±  27.873  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1           new     prepared  object  thrpt   20   207.810 ±  24.750  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc         reuse       normal  object  thrpt   20   217.180 ±  18.722  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc         reuse     prepared  object  thrpt   20   207.819 ±  23.562  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc           new       normal  object  thrpt   20   214.801 ±  28.415  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc           new     prepared  object  thrpt   20   216.493 ±  20.909  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1         reuse       normal  object  thrpt   20  1087.584 ±  88.325  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1         reuse     prepared  object  thrpt   20  1115.086 ± 103.262  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1           new       normal  object  thrpt   20  1111.623 ±  99.610  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1           new     prepared  object  thrpt   20  1065.311 ±  76.128  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc         reuse       normal  object  thrpt   20   715.300 ±  67.225  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc         reuse     prepared  object  thrpt   20   753.368 ±  50.600  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc           new       normal  object  thrpt   20   678.829 ±  81.305  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc           new     prepared  object  thrpt   20   683.853 ±  77.931  ops/s

github-actions · 2022-01-24T06:42:55Z

Benchmark                                 (client)  (connection)  (statement)  (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc1         reuse       normal  object  thrpt   20   217.486 ±  24.092  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1         reuse     prepared  object  thrpt   20   214.531 ±  24.273  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1           new       normal  object  thrpt   20   213.368 ±  23.031  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1           new     prepared  object  thrpt   20   212.004 ±  23.038  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc         reuse       normal  object  thrpt   20   219.993 ±  25.482  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc         reuse     prepared  object  thrpt   20   223.826 ±  26.966  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc           new       normal  object  thrpt   20   221.079 ±  24.768  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc           new     prepared  object  thrpt   20   221.584 ±  24.142  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1         reuse       normal  object  thrpt   20  1342.372 ±  94.688  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1         reuse     prepared  object  thrpt   20  1341.758 ± 109.327  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1           new       normal  object  thrpt   20  1335.903 ± 110.250  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1           new     prepared  object  thrpt   20  1357.388 ±  95.751  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc         reuse       normal  object  thrpt   20   881.557 ±  53.238  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc         reuse     prepared  object  thrpt   20   810.222 ±  76.477  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc           new       normal  object  thrpt   20   743.756 ±  62.011  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc           new     prepared  object  thrpt   20   830.568 ±  70.877  ops/s

github-actions · 2022-01-24T14:51:03Z

Benchmark                                 (client)  (connection)  (statement)  (type)   Mode  Cnt     Score    Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc1         reuse       normal  object  thrpt   20   201.640 ± 22.888  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1         reuse     prepared  object  thrpt   20   203.765 ± 24.698  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1           new       normal  object  thrpt   20   201.649 ± 26.472  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1           new     prepared  object  thrpt   20   201.527 ± 23.686  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc         reuse       normal  object  thrpt   20   210.149 ± 18.274  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc         reuse     prepared  object  thrpt   20   212.332 ± 22.137  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc           new       normal  object  thrpt   20   211.663 ± 26.317  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc           new     prepared  object  thrpt   20   209.550 ± 23.070  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1         reuse       normal  object  thrpt   20  1016.752 ± 96.928  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1         reuse     prepared  object  thrpt   20  1063.180 ± 57.560  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1           new       normal  object  thrpt   20  1031.810 ± 72.690  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1           new     prepared  object  thrpt   20  1045.367 ± 60.911  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc         reuse       normal  object  thrpt   20   678.027 ± 45.675  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc         reuse     prepared  object  thrpt   20   667.163 ± 69.436  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc           new       normal  object  thrpt   20   614.581 ± 50.957  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc           new     prepared  object  thrpt   20   682.945 ± 73.751  ops/s

github-actions · 2022-01-24T14:53:35Z

Benchmark                                 (client)  (connection)  (statement)  (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc1         reuse       normal  object  thrpt   20   209.885 ±  22.759  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1         reuse     prepared  object  thrpt   20   209.797 ±  24.329  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1           new       normal  object  thrpt   20   207.236 ±  27.008  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc1           new     prepared  object  thrpt   20   211.582 ±  24.292  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc         reuse       normal  object  thrpt   20   206.755 ±  29.760  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc         reuse     prepared  object  thrpt   20   217.701 ±  22.935  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc           new       normal  object  thrpt   20   215.399 ±  26.640  ops/s
Basic.insertOneRandomNumber   clickhouse-grpc-jdbc           new     prepared  object  thrpt   20   216.043 ±  26.944  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1         reuse       normal  object  thrpt   20  1083.612 ±  83.767  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1         reuse     prepared  object  thrpt   20  1027.421 ± 102.281  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1           new       normal  object  thrpt   20  1051.678 ±  84.544  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc1           new     prepared  object  thrpt   20  1070.535 ±  81.369  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc         reuse       normal  object  thrpt   20   683.292 ±  61.748  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc         reuse     prepared  object  thrpt   20   714.334 ±  61.043  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc           new       normal  object  thrpt   20   651.629 ±  55.758  ops/s
Basic.selectOneRandomNumber   clickhouse-grpc-jdbc           new     prepared  object  thrpt   20   720.645 ±  76.726  ops/s

tangjiangling · 2022-01-25T05:38:31Z

Fixed by #815. Now we can close this.

zhicwu changed the base branch from master to develop January 16, 2022 23:29

tangjiangling force-pushed the fix-wrong-decimal-precision branch from f8cd9e4 to e916539 Compare January 17, 2022 04:09

tangjiangling force-pushed the fix-wrong-decimal-precision branch from e916539 to ab51406 Compare January 23, 2022 21:26

tangjiangling mentioned this pull request Jan 23, 2022

Replace deprecated driver ru.yandex.clickhouse.ClickHouseDriver trinodb/trino#10559

Closed

3 tasks

tangjiangling force-pushed the fix-wrong-decimal-precision branch from ab51406 to a564b7b Compare January 24, 2022 06:25

tangjiangling force-pushed the fix-wrong-decimal-precision branch from a564b7b to 22a36ca Compare January 24, 2022 14:33

zhicwu and others added 5 commits January 24, 2022 22:35

Fix null DECIMAL_DIGITS and CHAR_OCTET_LENGTH

8ff064d

Fix test failure

9250280

Fix null DECIMAL_DIGITS for Decimal*/Datetime/Datetime64

ae8174b

Fix typo in ClickHouseDatabaseMetaDataTest

0959f36

Fix unquoted binary String

6ee8885

tangjiangling force-pushed the fix-wrong-decimal-precision branch from 22a36ca to 6ee8885 Compare January 24, 2022 14:36

tangjiangling mentioned this pull request Jan 25, 2022

Fix incorrect precision and scale from DatabaseMetaData.getColumns #815

Merged

tangjiangling closed this Jan 25, 2022

tangjiangling deleted the fix-wrong-decimal-precision branch January 25, 2022 05:38

Conversation

tangjiangling commented Jan 16, 2022

Uh oh!

CLAassistant commented Jan 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tangjiangling commented Jan 16, 2022

Uh oh!

tangjiangling commented Jan 16, 2022

Uh oh!

github-actions bot commented Jan 16, 2022

Uh oh!

tangjiangling commented Jan 16, 2022

Uh oh!

github-actions bot commented Jan 16, 2022

Uh oh!

zhicwu commented Jan 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tangjiangling commented Jan 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tangjiangling commented Jan 17, 2022

Uh oh!

tangjiangling commented Jan 17, 2022

Uh oh!

github-actions bot commented Jan 17, 2022

Uh oh!

zhicwu commented Jan 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tangjiangling commented Jan 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tangjiangling commented Jan 23, 2022

Uh oh!

github-actions bot commented Jan 23, 2022

Uh oh!

github-actions bot commented Jan 24, 2022

Uh oh!

github-actions bot commented Jan 24, 2022

Uh oh!

github-actions bot commented Jan 24, 2022

Uh oh!

tangjiangling commented Jan 25, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CLAassistant commented Jan 16, 2022 •

edited

Loading

zhicwu commented Jan 16, 2022 •

edited

Loading

tangjiangling commented Jan 17, 2022 •

edited

Loading

zhicwu commented Jan 17, 2022 •

edited

Loading

tangjiangling commented Jan 17, 2022 •

edited

Loading