Skip to content

[WIP]: Fix drop void partition column in iceberg connector#15740

Closed
krvikash wants to merge 1 commit intotrinodb:masterfrom
krvikash:fix-iceberg-insert-after-drop-void-partition-column
Closed

[WIP]: Fix drop void partition column in iceberg connector#15740
krvikash wants to merge 1 commit intotrinodb:masterfrom
krvikash:fix-iceberg-insert-after-drop-void-partition-column

Conversation

@krvikash
Copy link
Contributor

Description

IcebergMetadata#dropColumn does not remove the partition column from the current partition spec. which causes the failure when trying to select/insert/update on the table.

Partial fixes #15738

This PR depends on #15729 to get fixed completely.

Release notes

( ) This is not user-visible or docs only and no release notes are required.
(X) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

@cla-bot cla-bot bot added the cla-signed label Jan 17, 2023
@krvikash krvikash self-assigned this Jan 17, 2023
@krvikash
Copy link
Contributor Author

CI Failure: #15729 needs to be fixed.

2023-01-17T11:04:39.2590850Z java.lang.NullPointerException: Type cannot be null
2023-01-17T11:04:39.2591584Z 	at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:907)
2023-01-17T11:04:39.2592258Z 	at org.apache.iceberg.types.Types$NestedField.<init>(Types.java:446)
2023-01-17T11:04:39.2592777Z 	at org.apache.iceberg.types.Types$NestedField.optional(Types.java:415)
2023-01-17T11:04:39.2593328Z 	at org.apache.iceberg.PartitionSpec.partitionType(PartitionSpec.java:132)
2023-01-17T11:04:39.2593904Z 	at org.apache.iceberg.util.PartitionSet.lambda$new$0(PartitionSet.java:45)
2023-01-17T11:04:39.2594647Z 	at org.apache.iceberg.relocated.com.google.common.collect.RegularImmutableMap.forEach(RegularImmutableMap.java:292)
2023-01-17T11:04:39.2595343Z 	at org.apache.iceberg.util.PartitionSet.<init>(PartitionSet.java:45)
2023-01-17T11:04:39.2595876Z 	at org.apache.iceberg.util.PartitionSet.create(PartitionSet.java:37)
2023-01-17T11:04:39.2596436Z 	at org.apache.iceberg.ManifestFilterManager.<init>(ManifestFilterManager.java:94)
2023-01-17T11:04:39.2597040Z 	at org.apache.iceberg.MergingSnapshotProducer$DataFileFilterManager.<init>(MergingSnapshotProducer.java:971)
2023-01-17T11:04:39.2597748Z 	at org.apache.iceberg.MergingSnapshotProducer$DataFileFilterManager.<init>(MergingSnapshotProducer.java:969)
2023-01-17T11:04:39.2598574Z 	at org.apache.iceberg.MergingSnapshotProducer.<init>(MergingSnapshotProducer.java:122)
2023-01-17T11:04:39.2599069Z 	at org.apache.iceberg.MergeAppend.<init>(MergeAppend.java:32)
2023-01-17T11:04:39.2599521Z 	at org.apache.iceberg.BaseTransaction.newAppend(BaseTransaction.java:151)
2023-01-17T11:04:39.2600115Z 	at io.trino.plugin.iceberg.IcebergMetadata.finishInsert(IcebergMetadata.java:817)
2023-01-17T11:04:39.2629760Z 	at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.finishInsert(ClassLoaderSafeConnectorMetadata.java:519)
2023-01-17T11:04:39.2630481Z 	at io.trino.metadata.MetadataManager.finishInsert(MetadataManager.java:911)
2023-01-17T11:04:39.2631860Z 	at io.trino.sql.planner.LocalExecutionPlanner.lambda$createTableFinisher$4(LocalExecutionPlanner.java:4090)
2023-01-17T11:04:39.2632455Z 	at io.trino.operator.TableFinishOperator.getOutput(TableFinishOperator.java:319)
2023-01-17T11:04:39.2632938Z 	at io.trino.operator.Driver.processInternal(Driver.java:411)
2023-01-17T11:04:39.2633362Z 	at io.trino.operator.Driver.lambda$process$10(Driver.java:314)
2023-01-17T11:04:39.2633818Z 	at io.trino.operator.Driver.tryWithLock(Driver.java:706)
2023-01-17T11:04:39.2634196Z 	at io.trino.operator.Driver.process(Driver.java:306)
2023-01-17T11:04:39.2634603Z 	at io.trino.operator.Driver.processForDuration(Driver.java:277)
2023-01-17T11:04:39.2635226Z 	at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:752)
2023-01-17T11:04:39.2635821Z 	at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:165)
2023-01-17T11:04:39.2636578Z 	at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:519)
2023-01-17T11:04:39.2637054Z 	at io.trino.$gen.Trino_testversion____20230117_105655_175.run(Unknown Source)
2023-01-17T11:04:39.2637532Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2023-01-17T11:04:39.2638083Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2023-01-17T11:04:39.2638518Z 	at java.base/java.lang.Thread.run(Thread.java:833)
2023-01-17T11:04:39.2638710Z 
2023-01-17T11:04:39.2638718Z 

@github-actions
Copy link

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

@github-actions github-actions bot added the stale label Feb 20, 2023
@bitsondatadev
Copy link
Member

@krvikash are you still working on this? If you want someone to take a look, you'll need to fix conflicts and remove the WIP. If you need more time to work on this, we may just close this and you can reopen it when you have time to work on it. Let me know.

@krvikash
Copy link
Contributor Author

krvikash commented Feb 21, 2023

Hi @bitsondatadev, This PR depends on #15729 which further depends on apache/iceberg#4602. I am waiting for a fix from Iceberg.

@bitsondatadev
Copy link
Member

Let's close this until it's ready to be worked on. You can reopen this same one once the blockers are removed, we just want to only keep active non-blocked PRs open. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

Fails to insert into iceberg table after dropping a void transform partition column

2 participants