Skip to content

Retry Spark query when minReplication error happened#13743

Merged
ebyhr merged 2 commits intotrinodb:masterfrom
ebyhr:ebi/iceberg-spark-retry
Aug 23, 2022
Merged

Retry Spark query when minReplication error happened#13743
ebyhr merged 2 commits intotrinodb:masterfrom
ebyhr:ebi/iceberg-spark-retry

Conversation

@ebyhr
Copy link
Copy Markdown
Member

@ebyhr ebyhr commented Aug 19, 2022

Description

Fixes #13601

Documentation

(x) No documentation is needed.

Release notes

(x) No release notes entries required.

@ebyhr
Copy link
Copy Markdown
Member Author

ebyhr commented Aug 19, 2022

could only be written to 0 of the 1 minReplication nodes happened with Unsupported table change: Self-suppression not permitted, not Writing job aborted error.

https://github.com/trinodb/trino/runs/7911385034

@ebyhr ebyhr force-pushed the ebi/iceberg-spark-retry branch from c9c3a44 to d49704b Compare August 19, 2022 04:55
@ebyhr ebyhr changed the title Retry Spark query when Writing job aborted happened Retry Spark query when minReplication error happened Aug 19, 2022
@ebyhr
Copy link
Copy Markdown
Member Author

ebyhr commented Aug 19, 2022

io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.sql.AnalysisException: Can not rename the managed table('`default`.`test_trino_reading_migrated_nested_data_8wla2ykl5pms_BACKUP_`'). The associated location('hdfs://hadoop-master:9000/user/hive/warehouse/test_trino_reading_migrated_nested_data_8wla2ykl5pms') already exists.

https://github.com/trinodb/trino/runs/7915590576

@ebyhr ebyhr force-pushed the ebi/iceberg-spark-retry branch from bef0d9f to e93498d Compare August 22, 2022 06:09
@ebyhr ebyhr marked this pull request as ready for review August 22, 2022 08:35
@ebyhr ebyhr force-pushed the ebi/iceberg-spark-retry branch from e93498d to 98dd8e3 Compare August 22, 2022 09:35
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That may depend on Hadoop client version bundled.
That's probably whyb io.trino.tests.product.hive.HiveProductTest#ERROR_COMMITTING_WRITE_TO_HIVE_MATCH is an OR.

What about reusing that pattern, perhaps moving it out of the Hive-specific class (into some Hadoop-related class)?

@ebyhr ebyhr force-pushed the ebi/iceberg-spark-retry branch from 98dd8e3 to 191dd7f Compare August 22, 2022 23:56
@ebyhr ebyhr force-pushed the ebi/iceberg-spark-retry branch from 191dd7f to 17b07dc Compare August 23, 2022 00:33
@ebyhr ebyhr merged commit 9b7a7bf into trinodb:master Aug 23, 2022
@ebyhr ebyhr deleted the ebi/iceberg-spark-retry branch August 23, 2022 02:22
@github-actions github-actions bot added this to the 394 milestone Aug 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

TestIcebergSparkCompatibility test methods became flaky recently

2 participants