Skip to content

Propagate correct partition count for insert stages#25068

Merged
sopel39 merged 2 commits intotrinodb:masterfrom
sopel39:ks/split
Feb 19, 2025
Merged

Propagate correct partition count for insert stages#25068
sopel39 merged 2 commits intotrinodb:masterfrom
sopel39:ks/split

Conversation

@sopel39
Copy link
Copy Markdown
Member

@sopel39 sopel39 commented Feb 18, 2025

Previously optional partition count was passed to NodePartitioningManager. When partition count was unspecified, then maxHashPartitionCount was used. However, maxHashPartitionCount should only apply to intermediate stages, rather than insert stages. Insert stages are controlled via maxWriterTaskCount.

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## Iceberg
* Make engine honor `task.max-writer-count` for Iceberg partitioned writes. ({issue}`issuenumber`)

@cla-bot cla-bot bot added the cla-signed label Feb 18, 2025
@github-actions github-actions bot added the iceberg Iceberg connector label Feb 18, 2025
Previously optional partition count was passed to NodePartitioningManager.
When partition count was unspecified, then maxHashPartitionCount was used.
However, maxHashPartitionCount should only apply to intermediate stages, rather
than insert stages. Insert stages are controlled via maxWriterTaskCount.
@sopel39 sopel39 merged commit 0d8a46c into trinodb:master Feb 19, 2025
@sopel39 sopel39 deleted the ks/split branch February 19, 2025 17:50
@github-actions github-actions bot added this to the 471 milestone Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

2 participants