Skip to content

Implement dynamic filtering for JDBC connectors#13334

Merged
raunaqmorarka merged 3 commits intotrinodb:masterfrom
raunaqmorarka:jdbc-df
Aug 4, 2022
Merged

Implement dynamic filtering for JDBC connectors#13334
raunaqmorarka merged 3 commits intotrinodb:masterfrom
raunaqmorarka:jdbc-df

Conversation

@raunaqmorarka
Copy link
Copy Markdown
Member

@raunaqmorarka raunaqmorarka commented Jul 25, 2022 •
edited
Loading

Is this change a fix, improvement, new feature, refactoring, or other?

new feature

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

JDBC connectors

How would you describe this change to a non-technical end user or system administrator?

Improved performance for selective joins with JDBC source on the probe side

Related issues, pull requests, and links

Fixes #7968

Documentation

( ) No documentation is needed.
(x) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

( ) No release notes entries required.
(x) Release notes entries required with the following suggested text:

# ClickHouse, Druid, MariaDb, MySql, Oracle, PostgreSQL, Redshift, SingleStore, SQL server, Phoenix
* Improved performance for selective joins with JDBC source on the probe side. ({issue}`13334`)

@cla-bot cla-bot bot added the cla-signed label Jul 25, 2022
This was referenced Jul 25, 2022
Closed
Closed
@raunaqmorarka raunaqmorarka force-pushed the jdbc-df branch 2 times, most recently from bf56406 to f4af706 Compare July 26, 2022 04:29
@raunaqmorarka raunaqmorarka requested a review from lhofhansl July 26, 2022 04:30
@raunaqmorarka raunaqmorarka force-pushed the jdbc-df branch 2 times, most recently from 18bcd79 to 2638069 Compare July 26, 2022 08:13
@sopel39 sopel39 requested a review from skrzypo987 July 26, 2022 08:25
@raunaqmorarka raunaqmorarka force-pushed the jdbc-df branch 3 times, most recently from e6eb16e to 9734f49 Compare July 27, 2022 10:39
sopel39
sopel39 reviewed Jul 27, 2022
View reviewed changes
@raunaqmorarka raunaqmorarka requested a review from sopel39 July 28, 2022 09:35
@lhofhansl
Copy link
Copy Markdown
Member

lhofhansl commented Jul 30, 2022 •
edited
Loading

Tried it. Works well.
Test-failures seem relevant, though.

@raunaqmorarka raunaqmorarka force-pushed the jdbc-df branch 3 times, most recently from 662139c to 19bb03f Compare August 1, 2022 12:05
sopel39
sopel39 reviewed Aug 1, 2022
View reviewed changes
plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/JdbcColumnHandle.java Outdated
Copy link
Copy Markdown
Member

@sopel39 sopel39 Aug 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: separate commit

@raunaqmorarka raunaqmorarka force-pushed the jdbc-df branch 2 times, most recently from e113f40 to 00ae921 Compare August 1, 2022 20:49
@raunaqmorarka raunaqmorarka requested a review from sopel39 August 1, 2022 20:50
@raunaqmorarka raunaqmorarka mentioned this pull request Aug 2, 2022
Closed
sopel39
sopel39 reviewed Aug 3, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@sopel39 sopel39 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm % comments

sopel39
sopel39 approved these changes Aug 3, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@sopel39 sopel39 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

% comments

@sopel39
Copy link
Copy Markdown
Member

sopel39 commented Aug 3, 2022

Failed due #13362

raunaqmorarka and others added 3 commits August 4, 2022 08:37
@raunaqmorarka @sopel39
Implement dynamic filtering for JDBC connectors
74dbaf6 
Added JDBC catalog properties dynamic-filtering.enabled and
dynamic-filtering.wait-timeout to configure pushdown of
dynamic filters to JDBC queries.

Co-authored-by: Karol Sobczak <napewnotrafi@gmail.com>
@raunaqmorarka
Implement pushdown of dynamic filters in phoenix connector
1519fd3
@raunaqmorarka
Add domain-compaction-threshold to JDBC docs
fc1f0d1
@raunaqmorarka
Copy link
Copy Markdown
Member Author

raunaqmorarka commented Aug 4, 2022

JDBC DF postgresql sf10 partitioned.pdf
Over 3x improvement in TPCDS benchmark with data stored in PostgreSQL (sf10 partitioned schema).

@lhofhansl
Copy link
Copy Markdown
Member

lhofhansl commented Aug 4, 2022

With my tests (with an internal relational, analytical DB) I found the improvement to be more like 50x :)

@raunaqmorarka raunaqmorarka merged commit 9282a73 into trinodb:master Aug 4, 2022
@raunaqmorarka raunaqmorarka deleted the jdbc-df branch August 4, 2022 16:11
@raunaqmorarka raunaqmorarka mentioned this pull request Aug 4, 2022
Closed
@github-actions github-actions bot added this to the 393 milestone Aug 4, 2022
@colebow colebow mentioned this pull request Aug 5, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sopel39 sopel39 sopel39 approved these changes

@hashhar hashhar Awaiting requested review from hashhar

@findepi findepi Awaiting requested review from findepi

@skrzypo987 skrzypo987 Awaiting requested review from skrzypo987

+1 more reviewer

@lhofhansl lhofhansl lhofhansl left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla-signed docs performance

Milestone

393

Development

Successfully merging this pull request may close these issues.

Enable dynamic filtering in JDBC connector

3 participants

@raunaqmorarka @lhofhansl @sopel39