Skip to content

Conversation

@findinpath
Copy link
Contributor

@findinpath findinpath commented Oct 24, 2023

Description

Add tinyint/smallint/integer/bigint to double coercion for hive tables.

Enable the Hive users to retrieve the information from their tables after performing queries which change numeric integer columns to double.

Hive sample DDL query:

ALTER TABLE mytable CHANGE COLUMN mycolumn mycolumn double;

Additional context and related issues

This is a mere adaptation of #16869 to match timestamp to date coercion

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
() Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Hive
* Support tinyint/smallint/integer/bigint to double schema evolution. ({issue}`issuenumber`)

@cla-bot cla-bot bot added the cla-signed label Oct 24, 2023
@findinpath findinpath self-assigned this Oct 24, 2023
@findinpath findinpath added release-notes hive Hive connector labels Oct 24, 2023
@dain
Copy link
Member

dain commented Oct 24, 2023

Does the schema evolution match the behavior of Hive?

@dain
Copy link
Member

dain commented Oct 24, 2023

To answer my own question, I think this is the method that Hive uses to determine if a type schema type is allowed. The last lines of that code aren't really clear, but I believe it is says that anything on this list is coercible to anything higher on the list:

  1. PrimitiveCategory.BYTE
  2. PrimitiveCategory.SHORT
  3. PrimitiveCategory.INT
  4. PrimitiveCategory.LONG
  5. PrimitiveCategory.DECIMAL
  6. PrimitiveCategory.FLOAT
  7. PrimitiveCategory.DOUBLE
  8. PrimitiveCategory.STRING

@findinpath
Copy link
Contributor Author

Unrelated CI failures

io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: Query failed (#20231024_225456_00532_2pkqk): More than one latest metadata file found at location: hdfs://hadoop-master:9000/user/hive/warehouse/test_register_table_with_comments_parquet_vi7gmeoqbt-93efd287b14c4ac7a78f42f7bf0f227c/metadata, latest metadata files are [hdfs://hadoop-master:9000/user/hive/warehouse/test_register_table_with_comments_parquet_vi7gmeoqbt-93efd287b14c4ac7a78f42f7bf0f227c/metadata/00006-a44edbac-5800-4500-9100-bf0323ad38a8.metadata.json, hdfs://hadoop-master:9000/user/hive/warehouse/test_register_table_with_comments_parquet_vi7gmeoqbt-93efd287b14c4ac7a78f42f7bf0f227c/metadata/00006-cb7496dc-adca-4712-be81-27677393b54b.metadata.json]
	at io.trino.tempto.query.JdbcQueryExecutor.execute(JdbcQueryExecutor.java:119)

@dain
Copy link
Member

dain commented Oct 30, 2023

Is there an umbrella issue to track the missing coersions?

@Praveen2112 Praveen2112 force-pushed the findinpath/hive-integer-number-to-double-coercer branch from 1390e39 to 8c0ed85 Compare November 6, 2023 07:28
@findinpath
Copy link
Contributor Author

Is there an umbrella issue to track the missing coersions?

Created #19643 for providing of the past & ongoing schema evolution efforts in Hive.

@findinpath findinpath force-pushed the findinpath/hive-integer-number-to-double-coercer branch 2 times, most recently from 2c3014f to 01dc0a4 Compare November 7, 2023 05:03
@findinpath findinpath force-pushed the findinpath/hive-integer-number-to-double-coercer branch from 01dc0a4 to 936ddbe Compare November 7, 2023 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

3 participants