DOCS: describe type compatibility between Spark and Iceberg #1611
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -106,6 +106,8 @@ CREATE TABLE prod.db.sample ( | |
| USING iceberg | ||
| ``` | ||
|
|
||
| Iceberg will convert the column type in Spark to corresponding Iceberg type. Please check the section of [type compatibility on creating table](#spark-type-to-iceberg-type-on-creating-table) for details. | ||
|
|
||
| Table create commands, including CTAS and RTAS, support the full range of Spark create clauses, including: | ||
|
|
||
| * `PARTITION BY (partition-expressions)` to configure partitioning | ||
|
|
@@ -728,3 +730,82 @@ spark.read.format("iceberg").load("db.table.files").show(truncate = false) | |
| // Hadoop path table | ||
| spark.read.format("iceberg").load("hdfs://nn:8020/path/to/table#files").show(truncate = false) | ||
| ``` | ||
|
|
||
| ## Type compatibility | ||
|
|
||
| Spark and Iceberg support different set of types. Iceberg does the type conversion automatically, but not for all combinations, | ||
| so you may want to understand the type conversion in Iceberg in prior to design the types of columns in your tables. | ||
|
|
||
| ### Spark type to Iceberg type on creating table | ||
|
|
||
| This type conversion table describes how Spark types are converted to the Iceberg types. The conversion applies on creating Iceberg table via Spark without using Iceberg core API. | ||
|
|
||
| | Spark | Iceberg | Notes | | ||
| |-----------------|-------------------------|-------| | ||
| | boolean | boolean | | | ||
| | integer | integer | | | ||
| | short | integer | | | ||
| | byte | integer | | | ||
| | long | long | | | ||
| | float | float | | | ||
| | double | double | | | ||
| | date | date | | | ||
| | timestamp | timestamp with timezone | | | ||
| | string | string | | | ||
| | char | string | | | ||
| | varchar | string | | | ||
| | binary | binary | | | ||
| | decimal | decimal | | | ||
| | struct | struct | | | ||
| | array | list | | | ||
| | map | map | | | ||
|
|
||
| The type conversion is asymmetric: this table doesn't represent the types of Iceberg Spark can "read" from, or "write" to. | ||
| The following sections describe the feasibility on read/write for Iceberg type from Spark. | ||
|
|
||
| ### Iceberg to Spark on reading from Iceberg table | ||
|
|
||
| | Iceberg | Spark | Note | | ||
| |----------------------------|-------------------------|-------| | ||
| | boolean | boolean | | | ||
| | integer | integer | | | ||
| | long | long | | | ||
| | float | float | | | ||
| | double | double | | | ||
| | date | date | | | ||
| | time | N/A | | | ||
| | timestamp with timezone | timestamp | | | ||
| | timestamp without timezone | N/A | | | ||
|
Contributor
There was a problem hiding this comment. Should we convert timestamp without zone to timestamp in Spark when reading?
Contributor
Author
There was a problem hiding this comment. Honestly I wasn't too concerned about timestamp (I tried my best to consider the type as epoch based on UTC) so no strong opinion here. If we'd like to also support this (probably adjusting to the TZ in Spark?) it might be also good to do vice versa, so that read and write are symmetric. |
||
| | string | string | | | ||
| | uuid | string | | | ||
|
Contributor
Author
There was a problem hiding this comment. While I simply put uuid -> string based on the implementation of TypeToSparkType, I failed to verify this as it looks to be really tricky to write UUID column. It doesn't look possible to write UUID column from Spark. Even adding UUID type to SUPPORTED_PRIMITIVES in DataTest leads multiple tests failing. Is there any known way to write UUID column, or it'd be better to simply remove uuid here?
Contributor
There was a problem hiding this comment. We should ignore UUID for now. No engines support it and we are considering whether we should remove it from the spec.
Member
There was a problem hiding this comment. Followed up at trinodb/trino#6663 and on the mailing list. |
||
| | fixed | binary | | | ||
| | binary | binary | | | ||
| | decimal | decimal | | | ||
| | struct | struct | | | ||
| | list | array | | | ||
| | map | map | | | ||
|
|
||
| ### Spark type to Iceberg type on writing to Iceberg table | ||
|
|
||
| Note that the type conversion is a bit different from the one for creating table, as the target Iceberg table may have Iceberg types which aren't available on creating table via Spark. | ||
|
|
||
| | Spark | Iceberg | Note | | ||
| |-------------------------|-------------------------|-------------------------------------------| | ||
| | boolean | boolean | | | ||
| | integer | integer | numeric* | | ||
| | long | long | numeric* | | ||
| | float | float | numeric* | | ||
| | double | double | numeric* | | ||
| | date | date | | | ||
| | timestamp | timestamp with timezone | | | ||
| | string | string | | | ||
| | binary | fixed | assertion on the length will be performed | | ||
| | binary | binary | | | ||
| | decimal | decimal | numeric* | | ||
| | struct | struct | | | ||
| | array | list | | | ||
| | map | map | | | ||
|
|
||
| *numeric: can store value from another numeric types (including byte, short as well) | ||
|
|
||
| You can't write the column from Spark for missing Iceberg types on the right side of the table, like time and timestamp without timezone, and uuid. | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Binary can be written to a fixed column. It will be validated at write time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see, you've created two separate tables. I think I would probably combine them with notes about create vs write instead of having two. That would be more confusing because readers would need to choose which one they need.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I had to split read and write because of UUID, but if we don't mind about UUID then it'll be closely symmetric. I'll try to consolidate twos into one.
I still think we'd better having two different matrixes for create table and read/write (consolidated, Iceberg types to (->) Spark types), because in many cases the type to pivot on create table in Spark is Spark type (as these types are what end users need to type in), while the type to pivot on read/write table in any engines is Iceberg type. I thought the type to pivot on write to table is engine's column type but I changed my mind as the types of columns are Iceberg types hence end users need to think based on these types.
It might be arguable that end users need to think about the final type of the column (Iceberg type) when creating table, which might end up with pivoting Iceberg type on create table. I don't have a strong opinion, as it's also a valid opinion, but there might be also someone who wants to see the Iceberg table as Spark's world of view (restrict the usage to Spark only) and don't want to concern about Iceberg types.
Either of the direction would be fine for me. WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that having multiple tables causes users to need to look carefully for what is different between them, and increases what they need to pay attention to ("which table do I need?"). I'd like to have as few of these as possible. So I'd remove UUID and add notes for any differences between create and write.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK thanks for the opinion. I consolidated tables on create and write. Actually I tried to consolidate all of three tables, but it seemed a bit confusing as directions on conversion are opposite. Please take a look again. Thanks!