Skip to content

Conversation

@SemionPar
Copy link
Contributor

@SemionPar SemionPar commented Jan 23, 2023

Description

Use smaller tables or smaller data sets in general, where appropriate. This should speed up tests for some connectors considerably.

Additional context and related issues

Release notes

(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

@ebyhr
Copy link
Member

ebyhr commented Jan 24, 2023

Could you confirm CI failures?

@SemionPar SemionPar force-pushed the semionpar/downscale-base-connector-tests branch 3 times, most recently from bbae176 to 347cbd2 Compare January 24, 2023 10:54
@SemionPar SemionPar force-pushed the semionpar/downscale-base-connector-tests branch 4 times, most recently from 65ef90b to afe551f Compare January 27, 2023 12:01
@anusudarsan
Copy link
Member

@SemionPar can we mark this ready for review?

@SemionPar SemionPar marked this pull request as ready for review February 1, 2023 13:48
@SemionPar SemionPar force-pushed the semionpar/downscale-base-connector-tests branch 2 times, most recently from 6bc0d42 to 9ca9691 Compare February 1, 2023 14:09
@SemionPar SemionPar requested review from raunaqmorarka and removed request for hashhar and wendigo February 1, 2023 14:09
Copy link
Member

@hashhar hashhar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % questions

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where does 111_200 come from? the estimated join size was 83_400 AFAIK.

Copy link
Contributor Author

@SemionPar SemionPar Feb 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what I would expect, too, after consistently decreasing data size 10x, but actually its 111200 (joinDataSize when I put the breakpoint in

if (joinDataSize < getJoinPushdownAutomaticJoinToTablesRatio(session) * (leftDataSize + rightDataSize)) {
), so I changed the numbers to match the reality and relaxed the ratio in test even more.

@SemionPar SemionPar force-pushed the semionpar/downscale-base-connector-tests branch from 9d35e73 to 24d9588 Compare February 3, 2023 12:07
@anusudarsan
Copy link
Member

@hashhar who can merge this PR? looks like comments are addressed

Use smaller tables or smaller datasets in general, where appropriate.
This should speed up tests for some connectors considerably.
@SemionPar SemionPar force-pushed the semionpar/downscale-base-connector-tests branch from 24d9588 to 51d48fb Compare February 7, 2023 08:37
@raunaqmorarka raunaqmorarka merged commit bba58cd into trinodb:master Feb 7, 2023
@github-actions github-actions bot added this to the 407 milestone Feb 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

5 participants