Prevent creating the table when the specified schema does not exist by krvikash · Pull Request #15788 · trinodb/trino

krvikash · 2023-01-19T18:45:42Z

Description

Release notes

(X) Release notes are required, with the following suggested text:

# Iceberg
* Prevent creating the table when the specified schema does not exist. ({issue}`15779`)

ebyhr · 2023-01-20T03:39:21Z

@krvikash Could you confirm CI failures?

findinpath · 2023-01-20T05:08:06Z

@krvikash would there a similar approach as in Delta Lake help in this case ?

trino/plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java

Lines 667 to 673 in 01a51b1

    
           public void createTable(ConnectorSession session, ConnectorTableMetadata tableMetadata, boolean ignoreExisting) 
        
           { 
        
               SchemaTableName schemaTableName = tableMetadata.getTable(); 
        
               String schemaName = schemaTableName.getSchemaName(); 
        
               String tableName = schemaTableName.getTableName(); 
        
               Database schema = metastore.getDatabase(schemaName).orElseThrow(() -> new SchemaNotFoundException(schemaName));

alexjo2144

Can we test this?

core/trino-main/src/main/java/io/trino/execution/CreateTableTask.java

findinpath · 2023-01-31T09:06:08Z

Failing tests on trino-iceberg relate to #15905

Pls push an empty commit when the above mentioned PR lands.

krvikash · 2023-02-01T06:30:05Z

Addressed comments.

ebyhr · 2023-02-01T06:58:47Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorSmokeTest.java

We would like to keep the smoke test as simple as possible. Can we move to BaseIcebergConnectorTest instead? Or do you have a specific reason to run this test in the smoke test class?

Yes, there is a specific reason to have this test in BaseIcebergConnectorSmokeTest. I wanted to test against ~~cloud storage~~ different metastore. BaseIcebergConnectorTest only covers the Local file system.

without this fix:
Using Hive metastore -> files are created and Exception is thrown Schema (.*) not found
Using Glue metastore -> files are created and Exception is thrown <“Database non_existing_schema_mjmeupcfxy not found. (Service: AWSGlue; Status Code: 400; Error Code: EntityNotFoundException; Request ID: 2225b446-d53f-48f5-9fe9-de4fd3e74e54; Proxy: null)“>

with this fix: Schema (.*) not found exception is thrown for all metastore.

BaseIcebergConnector does not verify for glue. So IMO BaseIcebergConnectorSmokeTest is best suited.

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorSmokeTest.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

...test/java/io/trino/plugin/iceberg/catalog/glue/TestIcebergGlueCatalogConnectorSmokeTest.java

...test/java/io/trino/plugin/iceberg/catalog/jdbc/TestIcebergJdbcCatalogConnectorSmokeTest.java

krvikash · 2023-02-02T07:16:30Z

Addressed comments.

findinpath · 2023-02-02T09:33:29Z

Build is red see ci / maven-checks

krvikash · 2023-02-03T07:06:48Z

Error: Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.3.0:analyze-only (default) on project trino-iceberg: Dependency problems found -> [Help 1]

Warning: Used undeclared dependencies found:
Warning: com.qubole.rubix:rubix-presto-shaded:jar:0.3.18:compile
Warning: Non-test scoped test only dependencies found:
Warning: com.qubole.rubix:rubix-presto-shaded:jar:0.3.18:compile

I think it's unrelated to my change. Retriggerd the CI build.

krvikash · 2023-02-06T04:54:56Z

Build is red see ci / maven-checks

Addressed. It was a shaded class import that was causing issues.

krvikash · 2023-02-06T06:22:24Z

CI is failing with the Flaky test

2023-02-06T05:46:29.8748259Z tests | 2023-02-06 11:31:29 INFO: FAILURE / io.trino.tests.product.kudu.TestKuduConnectoKerberosSmokeTest.kerberosAuthTicketExpiryTest (Groups: profile_specific_tests, kudu) took 5.4 seconds

#14441

ebyhr · 2023-02-10T02:16:26Z

/test-with-secrets sha=fe53676f991b788e14fdfd0cbb2eda81236aa6a7

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorSmokeTest.java

...test/java/io/trino/plugin/iceberg/catalog/jdbc/TestIcebergJdbcCatalogConnectorSmokeTest.java

github-actions · 2023-02-10T10:41:50Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4141358815

ebyhr · 2023-02-10T12:00:35Z

/test-with-secrets sha=6b2619c9795863e24e55c61f312c8514e47c4004

krvikash · 2023-02-10T20:49:40Z

Fixed CI failure.

github-actions · 2023-02-10T22:09:04Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4145615692

ebyhr · 2023-02-10T23:22:21Z

/test-with-secrets sha=e63cd92d7c334d6140d233d832a90dddeac7d10e

github-actions · 2023-02-11T02:24:55Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4148929977

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorSmokeTest.java

ebyhr · 2023-02-12T00:23:41Z

/test-with-secrets sha=8fa7ac9eea1281f24fbc7eb546a65ebfaabbabc2

github-actions · 2023-02-12T00:47:25Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4153825112

krvikash · 2023-02-12T09:49:53Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4153825112

Fixed it

Error:  Failures: 
Error:    TestIcebergGlueCatalogAccessOperations.testCreateTable:143->assertGlueMetastoreApiInvocations:429->assertGlueMetastoreApiInvocations:466 Expected: 
		1 fewer occurrences of GET_DATABASE
Error:    TestIcebergGlueCatalogAccessOperations.testCreateTableAsSelect:159->assertGlueMetastoreApiInvocations:429->assertGlueMetastoreApiInvocations:466 Expected: 
		1 fewer occurrences of GET_DATABASE
[INFO] 
Error:  Tests run: 198, Failures: 2, Errors: 0, Skipped: 0

ebyhr · 2023-02-12T22:42:56Z

/test-with-secrets sha=93d31bd217f6e503d7b037680942206bd120c2f7

github-actions · 2023-02-12T23:42:36Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4158597848

krvikash · 2023-02-13T05:50:57Z

CI Failure:
Exiting issue: #13199

io.trino.tests.product.deltalake.TestDeltaLakeAlterTableCompatibility.testCommentOnColumn (Groups: profile_specific_tests, delta-lake-databricks, delta-lake-oss) took 5.1 seconds

This is a new error (https://github.com/trinodb/trino/actions/runs/4158597848/jobs/7194014765). I don't think this error is related to this PR.

io.trino.tests.product.kafka.TestKafkaProtobuf.testInsertAllDataType (Groups: profile_specific_tests, kafka) took 0.8 seconds

cla-bot bot added the cla-signed label Jan 19, 2023

krvikash requested review from alexjo2144, ebyhr, findepi and findinpath January 19, 2023 18:51

krvikash force-pushed the fix-schema-does-not-exists-in-create-table branch from 1a1bc4c to 0f2e0e2 Compare January 27, 2023 06:29

alexjo2144 reviewed Jan 30, 2023

View reviewed changes

core/trino-main/src/main/java/io/trino/execution/CreateTableTask.java Outdated Show resolved Hide resolved

krvikash force-pushed the fix-schema-does-not-exists-in-create-table branch from 0f2e0e2 to e9ada98 Compare January 30, 2023 17:59

krvikash force-pushed the fix-schema-does-not-exists-in-create-table branch from 73d09d2 to c3fb2eb Compare February 1, 2023 06:16

ebyhr reviewed Feb 1, 2023

View reviewed changes

krvikash force-pushed the fix-schema-does-not-exists-in-create-table branch from c3fb2eb to 0f8b97e Compare February 1, 2023 09:52

findinpath reviewed Feb 2, 2023

View reviewed changes

krvikash force-pushed the fix-schema-does-not-exists-in-create-table branch from 0f8b97e to 9b98279 Compare February 2, 2023 07:16

krvikash changed the title ~~Fix Create Table when schema does not exist~~ Prevent creating the table when the specified schema does not exist Feb 2, 2023

krvikash force-pushed the fix-schema-does-not-exists-in-create-table branch from 9b98279 to 84a9eb2 Compare February 3, 2023 07:06

krvikash force-pushed the fix-schema-does-not-exists-in-create-table branch 2 times, most recently from d6be2a6 to fe53676 Compare February 6, 2023 04:52

krvikash self-assigned this Feb 7, 2023

findinpath approved these changes Feb 7, 2023

View reviewed changes

findinpath requested a review from ebyhr February 7, 2023 13:00

krvikash force-pushed the fix-schema-does-not-exists-in-create-table branch from c2f6ab6 to 1e97a1e Compare February 10, 2023 09:04

findinpath approved these changes Feb 10, 2023

View reviewed changes

ebyhr approved these changes Feb 10, 2023

View reviewed changes

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorSmokeTest.java Outdated Show resolved Hide resolved

...test/java/io/trino/plugin/iceberg/catalog/jdbc/TestIcebergJdbcCatalogConnectorSmokeTest.java Outdated Show resolved Hide resolved

krvikash force-pushed the fix-schema-does-not-exists-in-create-table branch 2 times, most recently from 8d483af to 1f50b86 Compare February 10, 2023 10:03

krvikash force-pushed the fix-schema-does-not-exists-in-create-table branch from 1f50b86 to 6b2619c Compare February 10, 2023 11:17

krvikash force-pushed the fix-schema-does-not-exists-in-create-table branch from 6b2619c to e63cd92 Compare February 10, 2023 20:48

ebyhr reviewed Feb 11, 2023

View reviewed changes

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorSmokeTest.java Outdated Show resolved Hide resolved

krvikash force-pushed the fix-schema-does-not-exists-in-create-table branch from e63cd92 to 8fa7ac9 Compare February 11, 2023 11:46

krvikash added 3 commits February 12, 2023 15:18

Rename schemaUrl to schemaPath in TestIcebergGcsConnectorSmokeTest

23ee4f2

Set jdbc warehouse dir property via setIcebergProperties for jdbc tests

958910c

Prevent creating the table when the specified schema does not exist

93d31bd

krvikash force-pushed the fix-schema-does-not-exists-in-create-table branch from 8fa7ac9 to 93d31bd Compare February 12, 2023 09:48

ebyhr merged commit 607decc into trinodb:master Feb 13, 2023

ebyhr mentioned this pull request Feb 13, 2023

Release notes for 407 #15854

Closed

krvikash deleted the fix-schema-does-not-exists-in-create-table branch February 13, 2023 06:20

github-actions bot added this to the 407 milestone Feb 13, 2023

colebow mentioned this pull request Feb 13, 2023

Add Trino 407 release notes #15919

Merged

Conversation

krvikash commented Jan 19, 2023 • edited by ebyhr Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Release notes

Uh oh!

ebyhr commented Jan 20, 2023

Uh oh!

findinpath commented Jan 20, 2023

Uh oh!

alexjo2144 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

findinpath commented Jan 31, 2023

Uh oh!

krvikash commented Feb 1, 2023

Uh oh!

ebyhr Feb 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krvikash Feb 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krvikash Feb 1, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

krvikash commented Feb 2, 2023

Uh oh!

findinpath commented Feb 2, 2023

Uh oh!

krvikash commented Feb 3, 2023

Uh oh!

krvikash commented Feb 6, 2023

Uh oh!

krvikash commented Feb 6, 2023

Uh oh!

ebyhr commented Feb 10, 2023

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 10, 2023

Uh oh!

ebyhr commented Feb 10, 2023

Uh oh!

krvikash commented Feb 10, 2023

Uh oh!

github-actions bot commented Feb 10, 2023

Uh oh!

ebyhr commented Feb 10, 2023

Uh oh!

github-actions bot commented Feb 11, 2023

Uh oh!

Uh oh!

ebyhr commented Feb 12, 2023

Uh oh!

github-actions bot commented Feb 12, 2023

Uh oh!

krvikash commented Feb 12, 2023

Uh oh!

ebyhr commented Feb 12, 2023

Uh oh!

github-actions bot commented Feb 12, 2023

Uh oh!

krvikash commented Feb 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

krvikash commented Jan 19, 2023 •

edited by ebyhr

Loading

ebyhr Feb 1, 2023 •

edited

Loading

krvikash Feb 1, 2023 •

edited

Loading

krvikash commented Feb 13, 2023 •

edited

Loading