Skip to content

Releases: MobileTeleSystems/spark-dialect-extension

0.0.2

02 Oct 13:30
@github-actions github-actions
0.0.2
4136bc1
This commit was signed with the committer’s verified signature. The key has expired.
dolfinus Maxim Martynov
GPG key ID: 4D63C82ED5B75257
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.0.2 Latest
Latest
  • Allow writing ArrayType(TimestampType()) Spark column as Clickhouse's Array(DateTime64(6)).
  • Allow writing ArrayType(ShortType()) Spark column as Clickhouse's Array(Int16).
Assets 4
Loading

0.0.1

01 Oct 09:46
@github-actions github-actions
0.0.1
d8a6f91
This commit was signed with the committer’s verified signature. The key has expired.
dolfinus Maxim Martynov
GPG key ID: 4D63C82ED5B75257
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.0.1

First release! 🎉

This version includes custom Clickhouse dialect for Apache Spark 3.5.x, with following enhancements:

  • support for writing Spark's ArrayType to Clickhouse. Currently only few types are supported, like ArrayType(StringType), ArrayType(ByteType), ArrayType(LongType), ArrayType(FloatType). Unfortunately, reading Arrays from Clickhouse to Spark is not fully supported for now.
  • fixed issue when writing Spark's TimestampType lead to creating Clickhouse table with DateTime64(0) instead of DateTime64(6), resulting a precision loss (fractions of seconds were dropped).
  • fixed issue when writing Spark's BooleanType lead to creating Clickhouse table with UInt64 column instead of Bool.
Loading