Skip to content

Conversation

satanson
Copy link
Contributor

@satanson satanson commented Sep 10, 2025

Why I'm doing:

Full sort using german string for comparison can accelerate queries if the string-type order-by column has different prefix 4 bytes.

use sv enable_full_sort_use_german_string to control this behavior, it is false in default.

for queries constructed on click_bench as follows

-- Q1
with cte as( select MobilePhoneModel, count(1) over(partition by MobilePhoneModel) as cnt from hits) 
select /*+SET_VAR(enable_full_sort_use_german_string=true)*/ sum(murmur_hash3_32(MobilePhoneModel, cnt)) from cte
-- Q2
with cte as( select URL, count(1) over(partition by URL) as cnt from hits) 
select /*+SET_VAR(enable_full_sort_use_german_string=true)*/ sum(murmur_hash3_32(URL, cnt)) from cte
-- Q3
with cte as( select SearchPhrase, count(1) over(partition by SearchPhrase) as cnt from hits) 
select /*+SET_VAR(enable_full_sort_use_german_string=true)*/ sum(murmur_hash3_32(SearchPhrase, cnt)) from cte
-- Q4
with cte as( select Title, count(1) over(partition by Title) as cnt from hits) 
select /*+SET_VAR(enable_full_sort_use_german_string=true)*/ sum(murmur_hash3_32(Title, cnt)) from cte
-- Q5
with cte as( select Referer, count(1) over(partition by Referer) as cnt from hits) 
select /*+SET_VAR(enable_full_sort_use_german_string=true)*/ sum(murmur_hash3_32(Referer, cnt)) from cte
-- Q6
with cte as( select MobilePhoneModel, SearchPhrase, count(1) over(partition by MobilePhoneModel, SearchPhrase) as cnt from hits) 
select /*+SET_VAR(enable_full_sort_use_german_string=true)*/ sum(murmur_hash3_32(MobilePhoneModel, SearchPhrase, cnt)) from cte
-- Q7
with cte as( select MobilePhoneModel, URL, count(1) over(partition by MobilePhoneModel, URL) as cnt from hits) 
select /*+SET_VAR(enable_full_sort_use_german_string=true)*/ sum(murmur_hash3_32(MobilePhoneModel, URL, cnt)) from cte

The test result
image

Performances of Q2, Q5, Q7 decrease significantly since the order-by column URL and Referer are almost the same.

What I'm doing:

Fixes #issue

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 4.0
    • 3.5
    • 3.4
    • 3.3

@satanson satanson requested a review from a team as a code owner September 10, 2025 08:08
@wanpengfei-git wanpengfei-git requested a review from a team September 10, 2025 08:08
@github-actions github-actions bot added the 4.0 label Sep 10, 2025
};

using Bytes = starrocks::raw::RawVectorPad16<uint8_t, ColumnAllocator<uint8_t>>;
class GermanStringBinaryColumnExternalAllocator : public GermanStringExternalAllocator {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems unused ?

return os;
}
union {
uint32_t len;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this len is not necessary?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no overhead

@satanson satanson force-pushed the fullsort_german_string branch from 3769f98 to a4fbc96 Compare September 10, 2025 08:37
@satanson satanson requested a review from a team as a code owner September 10, 2025 08:37
@satanson satanson force-pushed the fullsort_german_string branch from a4fbc96 to fd2bf2a Compare September 11, 2025 02:19
Copy link

[Java-Extensions Incremental Coverage Report]

pass : 0 / 0 (0%)

Copy link

[FE Incremental Coverage Report]

fail : 2 / 5 (40.00%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/qe/SessionVariable.java 2 5 40.00% [5326, 5327, 5330]

@satanson satanson force-pushed the fullsort_german_string branch from fd2bf2a to 42d38c8 Compare September 11, 2025 03:47
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants