Skip to content

Add coercion for timestamp in Delta Lake connector #19336

@findinpath

Description

@findinpath
trino> create schema delta.tiny with (location='s3://test-bucket/');
CREATE SCHEMA
trino> create table hive.tiny.hivetable (id integer, ts timestamp);
CREATE TABLE
trino> insert into hive.tiny.hivetable values (1, CURRENT_TIMESTAMP);
INSERT: 1 row

Query 20231010_143933_00004_rbgew, FINISHED, 1 node
http://localhost:8080/ui/query.html?20231010_143933_00004_rbgew
Splits: 8 total, 8 done (100.00%)
CPU Time: 0.3s total,     0 rows/s,     0B/s, 82% active
Per Node: 0.2 parallelism,     0 rows/s,     0B/s
Parallelism: 0.2
Peak Memory: 96KB
1.95 [0 rows, 0B] [0 rows/s, 0B/s]

trino> select * from hive.tiny.hivetable;
 id |           ts            
----+-------------------------
  1 | 2023-10-10 16:39:33.750 
(1 row)

Query 20231010_143941_00005_rbgew, FINISHED, 1 node
http://localhost:8080/ui/query.html?20231010_143941_00005_rbgew
Splits: 1 total, 1 done (100.00%)
CPU Time: 0.1s total,    13 rows/s, 4.84KB/s, 96% active
Per Node: 0.3 parallelism,     3 rows/s, 1.31KB/s
Parallelism: 0.3
Peak Memory: 176B
0.27 [1 rows, 362B] [3 rows/s, 1.31KB/s]

trino> create table delta.tiny.deltatable as select * from hive.tiny.hivetable;
Query 20231010_144009_00006_rbgew failed: Unsupported type: timestamp(3)
io.trino.spi.TrinoException: Unsupported type: timestamp(3)
	at io.trino.plugin.deltalake.transactionlog.DeltaLakeSchemaSupport.validatePrimitiveType(DeltaLakeSchemaSupport.java:393)
	at io.trino.plugin.deltalake.transactionlog.DeltaLakeSchemaSupport.validateType(DeltaLakeSchemaSupport.java:368)
	at io.trino.plugin.deltalake.transactionlog.DeltaLakeSchemaSupport.validateType(DeltaLakeSchemaSupport.java:359)
	at io.trino.plugin.deltalake.DeltaLakeMetadata.checkColumnTypes(DeltaLakeMetadata.java:1174)
	at io.trino.plugin.deltalake.DeltaLakeMetadata.validateTableColumns(DeltaLakeMetadata.java:1139)
	at io.trino.plugin.deltalake.DeltaLakeMetadata.getNewTableLayout(DeltaLakeMetadata.java:670)

Related effort for Iceberg

#13981

Metadata

Metadata

Assignees

Labels

delta-lakeDelta Lake connectorenhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions