Skip to content

Fix Partition Projection storage location#16349

Merged
findepi merged 1 commit intotrinodb:masterfrom
aczajkowski:acz/fix_parition_projection_storage_location
Mar 6, 2023
Merged

Fix Partition Projection storage location#16349
findepi merged 1 commit intotrinodb:masterfrom
aczajkowski:acz/fix_parition_projection_storage_location

Conversation

@aczajkowski
Copy link
Member

Description

Fix handling of partition_projection_location_template table property mapped from storage.location.template hive table property.

Additional context and related issues

When template contained upper case characters S3 List Objects operations was not returning any objects which resulted in 0 results queries. This was caused by lowercasing Trino table property during convertion to Hive table property.

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Section
* Fix handling of `partition_projection_location_template` table property when uppercase characters are used

@cla-bot cla-bot bot added the cla-signed label Mar 2, 2023
@findepi findepi removed their request for review March 2, 2023 16:37
Copy link
Member

@losipiuk losipiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. But I would prefer someone with more context to take a look. @findepi @findinpath ?

@aczajkowski aczajkowski force-pushed the acz/fix_parition_projection_storage_location branch from ec6590a to 00694b1 Compare March 2, 2023 17:08
@findepi findepi requested a review from findinpath March 2, 2023 17:43
@github-actions github-actions bot added hive Hive connector tests:hive labels Mar 2, 2023
@aczajkowski aczajkowski force-pushed the acz/fix_parition_projection_storage_location branch from 00694b1 to 02e9b08 Compare March 2, 2023 18:57
@aczajkowski aczajkowski force-pushed the acz/fix_parition_projection_storage_location branch 2 times, most recently from ab955f1 to 9076412 Compare March 4, 2023 14:32
@aczajkowski aczajkowski force-pushed the acz/fix_parition_projection_storage_location branch from 9076412 to c1489a4 Compare March 5, 2023 10:17
@aczajkowski
Copy link
Member Author

aczajkowski commented Mar 5, 2023

Seems only test job that is failing is testing\trino-tests
Test that where failing in different combinations:

  • TestDistributedSpilledQueries.testOrderByWithAgregation
  • TestDistributedSpilledQueries.testPredicate
  • TestSpilledJoinQueries.testSemiJoin
  • TestSpilledOrderByQueries.testOrderByWithOutputColumnReference

Similar problem can be seen on latest master builds
bf293f8
https://github.com/trinodb/trino/actions/runs/4334320154/jobs/7568155342

Root cause stack trace is always the same

Caused by: java.lang.IndexOutOfBoundsException: Index (0) is greater than or equal to list size (0)
	at it.unimi.dsi.fastutil.longs.LongArrayList.getLong(LongArrayList.java:267)
	at io.trino.operator.PagesIndex.buildPage(PagesIndex.java:306)
	at io.trino.operator.PagesIndex$2.computeNext(PagesIndex.java:631)
	at io.trino.operator.PagesIndex$2.computeNext(PagesIndex.java:624)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
	at io.trino.operator.WorkProcessorUtils.lambda$fromIterator$0(WorkProcessorUtils.java:103)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:413)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:347)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:413)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:347)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:413)
	at io.trino.operator.WorkProcessorUtils$2.process(WorkProcessorUtils.java:126)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:413)
	at io.trino.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:262)
	at io.trino.operator.WorkProcessorUtils$YieldingProcess.process(WorkProcessorUtils.java:182)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:413)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:347)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:413)
	at io.trino.operator.WorkProcessorUtils$YieldingIterator.computeNext(WorkProcessorUtils.java:81)
	at io.trino.operator.WorkProcessorUtils$YieldingIterator.computeNext(WorkProcessorUtils.java:67)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
	at io.trino.operator.OrderByOperator.getOutput(OrderByOperator.java:275)
	at io.trino.operator.Driver.processInternal(Driver.java:394)
	at io.trino.operator.Driver.lambda$process$8(Driver.java:297)
	at io.trino.operator.Driver.tryWithLock(Driver.java:689)
	at io.trino.operator.Driver.process(Driver.java:289)
	at io.trino.operator.Driver.processForDuration(Driver.java:260)
	at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:773)
	at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:165)
	at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:523)
	at io.trino.$gen.Trino_testversion____20230305_044615_29138.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

@aczajkowski
Copy link
Member Author

@electrum @findepi Thank you for review. All comments applied.
Should I create na issue for above problem ☝️

@findepi findepi merged commit 45a96cd into trinodb:master Mar 6, 2023
@github-actions github-actions bot added this to the 410 milestone Mar 6, 2023
@findepi
Copy link
Member

findepi commented Mar 7, 2023

Test that where failing in different combinations:

  • TestDistributedSpilledQueries.testOrderByWithAgregation
  • TestDistributedSpilledQueries.testPredicate
  • TestSpilledJoinQueries.testSemiJoin
  • TestSpilledOrderByQueries.testOrderByWithOutputColumnReference

Similar problem can be seen on latest master builds bf293f8 https://github.com/trinodb/trino/actions/runs/4334320154/jobs/7568155342

Root cause stack trace is always the same

Caused by: java.lang.IndexOutOfBoundsException: Index (0) is greater than or equal to list size (0)
	at it.unimi.dsi.fastutil.longs.LongArrayList.getLong(LongArrayList.java:267)

Any reason this is a comment and not a top-level issue?
i created #16406 for one of the problems reported, but don't know if this is one problem or multiple

@aczajkowski
Copy link
Member Author

Any reason this is a comment and not a top-level issue?
i created #16406 for one of the problems reported, but don't know if this is one problem or multiple

Non of I know. Just wanted to be sure so I asked here: #16349 (comment)

@electrum @findepi Thank you for review. All comments applied.
Should I create na issue for above problem ☝️

But I assume at this stage it's a must. So just reposted this comment in your Issue. #16406 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed hive Hive connector

Development

Successfully merging this pull request may close these issues.

4 participants