Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ arrow-memory-netty-buffer-patch/18.3.0//arrow-memory-netty-buffer-patch-18.3.0.j
arrow-memory-netty/18.3.0//arrow-memory-netty-18.3.0.jar
arrow-vector/18.3.0//arrow-vector-18.3.0.jar
audience-annotations/0.12.0//audience-annotations-0.12.0.jar
avro-ipc/1.12.0//avro-ipc-1.12.0.jar
avro-mapred/1.12.0//avro-mapred-1.12.0.jar
avro/1.12.0//avro-1.12.0.jar
avro-ipc/1.12.1//avro-ipc-1.12.1.jar
avro-mapred/1.12.1//avro-mapred-1.12.1.jar
avro/1.12.1//avro-1.12.1.jar
azure-data-lake-store-sdk/2.3.9//azure-data-lake-store-sdk-2.3.9.jar
azure-keyvault-core/1.0.0//azure-keyvault-core-1.0.0.jar
azure-storage/7.0.1//azure-storage-7.0.1.jar
Expand Down
4 changes: 2 additions & 2 deletions docs/sql-data-sources-avro.md
Original file line number Diff line number Diff line change
Expand Up @@ -477,7 +477,7 @@ Submission Guide for more details.
[adm]: submitting-applications.html#advanced-dependency-management

## Supported types for Avro -> Spark SQL conversion
Currently Spark supports reading all [primitive types](https://avro.apache.org/docs/1.12.0/specification/#primitive-types) and [complex types](https://avro.apache.org/docs/1.12.0/specification/#complex-types) under records of Avro.
Currently Spark supports reading all [primitive types](https://avro.apache.org/docs/1.12.1/specification/#primitive-types) and [complex types](https://avro.apache.org/docs/1.12.1/specification/#complex-types) under records of Avro.
<table>
<thead><tr><th><b>Avro type</b></th><th><b>Spark SQL type</b></th></tr></thead>
<tr>
Expand Down Expand Up @@ -541,7 +541,7 @@ In addition to the types listed above, it supports reading `union` types. The fo
3. `union(something, null)`, where something is any supported Avro type. This will be mapped to the same Spark SQL type as that of something, with nullable set to true.
All other union types are considered complex. They will be mapped to StructType where field names are member0, member1, etc., in accordance with members of the union. This is consistent with the behavior when converting between Avro and Parquet.

It also supports reading the following Avro [logical types](https://avro.apache.org/docs/1.12.0/specification/#logical-types):
It also supports reading the following Avro [logical types](https://avro.apache.org/docs/1.12.1/specification/#logical-types):

<table>
<thead><tr><th><b>Avro logical type</b></th><th><b>Avro type</b></th><th><b>Spark SQL type</b></th></tr></thead>
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@
-->
<codahale.metrics.version>4.2.33</codahale.metrics.version>
<!-- Should be consistent with SparkBuild.scala and docs -->
<avro.version>1.12.0</avro.version>
<avro.version>1.12.1</avro.version>
<aws.kinesis.client.version>1.15.3</aws.kinesis.client.version>
<!-- Should be consistent with Kinesis client dependency -->
<aws.java.sdk.version>1.12.681</aws.java.sdk.version>
Expand Down
2 changes: 1 addition & 1 deletion project/SparkBuild.scala
Original file line number Diff line number Diff line change
Expand Up @@ -1193,7 +1193,7 @@ object DependencyOverrides {
dependencyOverrides += "com.google.guava" % "guava" % guavaVersion,
dependencyOverrides ++= jacksonDeps.key.value,
dependencyOverrides += "jline" % "jline" % "2.14.6",
dependencyOverrides += "org.apache.avro" % "avro" % "1.12.0",
dependencyOverrides += "org.apache.avro" % "avro" % "1.12.1",
dependencyOverrides += "org.slf4j" % "slf4j-api" % "2.0.17")
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,14 +83,14 @@ private[sql] class AvroOptions(

/**
* Top level record name in write result, which is required in Avro spec.
* See https://avro.apache.org/docs/1.12.0/specification/#schema-record .
* See https://avro.apache.org/docs/1.12.1/specification/#schema-record .
* Default value is "topLevelRecord"
*/
val recordName: String = parameters.getOrElse(RECORD_NAME, "topLevelRecord")

/**
* Record namespace in write result. Default value is "".
* See Avro spec for details: https://avro.apache.org/docs/1.12.0/specification/#schema-record .
* See Avro spec for details: https://avro.apache.org/docs/1.12.1/specification/#schema-record .
*/
val recordNamespace: String = parameters.getOrElse(RECORD_NAMESPACE, "")

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -816,7 +816,7 @@ class HiveClientSuite(version: String) extends HiveVersionSuite(version) {
test("Decimal support of Avro Hive serde") {
val tableName = "tab1"
// TODO: add the other logical types. For details, see the link:
// https://avro.apache.org/docs/1.12.0/specification/#logical-types
// https://avro.apache.org/docs/1.12.1/specification/#logical-types
val avroSchema =
"""{
| "name": "test_record",
Expand Down