[SPARK-46612][SQL] Do not convert array type string retrieved from jdbc driver

[phanhuyn]

Hi, thanks for checking the PR. This is a small bug fix to make Scala Spark works with Clickhouse's array type. Let me know if this could cause problem on other DB types.

(Please help to trigger CI if possible. I failed to make the build pipeline run - any help is appreciated)

Why are the changes needed?

The PR is to fix issue describe at: ClickHouse/clickhouse-java#1505

When using spark to write an array of string to Clickhouse, the Clickhouse JDBC driver throws java.lang.IllegalArgumentException: Unknown data type: string exception.

The exception was due to Spark JDBC utils passing an invalid type value string (should be String). The original type values retrieved from Clickhouse JDBC is correct, but Spark JDBC utils attempts to convert to type string to lower case:

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala

Line 639 in 6b93153

.toLowerCase(Locale.ROOT).split("\\(")(0)

What changes were proposed in this pull request?

• Remove the lowercase cast. The string value retrieved from JDBC driver implementation should be passed as-is, Spark shouldn't try to modify the value.

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Follow the reproduce step at: JDBC throws java.lang.IllegalArgumentException: Unknown data type: string when write array string with Apache Spark scala ClickHouse/clickhouse-java#1505
  • Create ClickHouse table with array string type
  • Write scala Spark job to write to clickhouse
• Verify the change fixes the issue

Was this patch authored or co-authored using generative AI tooling?

No

[HyukjinKwon]

Thanks for the PR. Mind creating a JIRA please? (see also https://spark.apache.org/contributing.html).

[phanhuyn]

Thanks for the PR. Mind creating a JIRA please? (see also https://spark.apache.org/contributing.html).

Thanks for the reply @HyukjinKwon. I've applied for a JIRA account at https://selfserve.apache.org/jira-account.html, will be creating the story when the account is approved.

Update: JIRA is created https://issues.apache.org/jira/browse/SPARK-46612

[yaooqinn]

Hi @phanhuyn, Could you enable the GitHub action?

[phanhuyn]

Hi @phanhuyn, Could you enable the GitHub action?

I enabled GitHub action and triggered a few builds. It's failing at "Run documentation build" step. I'm not sure why this change would cause the documentation build to fail, could you help to take a look? Thank you.

The action url: https://github.com/phanhuyn/spark/actions/runs/7522708428/job/20479020431

[yaooqinn]

This build error shall be fixed, you can rebase the master branch and try again.

[yaooqinn]

It looks reasonable to me to respect the underlying database-specific type definition

[yaooqinn]

Thank you @phanhuyn. Merged to master

[phanhuyn]

thanks @yaooqinn

[yaooqinn]

Oops, I didn't pay much attention to the authorship resolution step in the merge script, the Lead-authored-by: shall be Nguyen Phan Huy

[phanhuyn]

Oops, I didn't pay much attention to the authorship resolution step in the merge script, the Lead-authored-by: shall be Nguyen Phan Huy

All good, no problem.