Skip to content

Pre-allocate capacity for definition & repetition levels#1

Merged
theosib-amazon merged 1 commit intomasterfrom
theosib-amazon-reserve-levels
Mar 25, 2022
Merged

Pre-allocate capacity for definition & repetition levels#1
theosib-amazon merged 1 commit intomasterfrom
theosib-amazon-reserve-levels

Conversation

@theosib-amazon
Copy link
Owner

Testing shows 5% to 15% performance improvement on a lot of queries by pre-reserving capacity for definitionLevels and repetitionLevels.

Description

I did profiling on Trino while doing TPCDS queries from a Parquet source and noticed that a lot of them were wasting time on array copy while growing capacity of the definitionLevels and definitionLevels integer array lists. Pre-reserving capacity sped up a lot of queries, some as much as 15%.

Is this change a fix, improvement, new feature, refactoring, or other?
Improvement

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)
Change to the parquet reader component

How would you describe this change to a non-technical end user or system administrator?
Speed up parquet column reading a bit by pre-reserving capacity in some of the containers.

Related issues, pull requests, and links

Performance optimization

Documentation

( ) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

( ) No release notes entries required.
( ) Release notes entries required with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

Testing shows 5% to 15% performance improvement on a lot of queries by pre-reserving capacity for definitionLevels and repetitionLevels.
@theosib-amazon theosib-amazon merged commit 5e5abc8 into master Mar 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant