Skip to content

Fix handing of grouped ordered aggregations with filtering #13627

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dain merged 1 commit into from
Aug 13, 2022
Merged

Fix handing of grouped ordered aggregations with filtering #13627

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion core/trino-main/src/main/java/io/trino/operator/aggregation/DistinctAccumulatorFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ public void addInput(GroupByIdBlock groupIdsBlock, Page page, Optional<Block> ma
columnIndexes[i] = i + 1;
}
Page filtered = filteredWithGroup.getColumns(columnIndexes);

// NOTE: the accumulator must be called even if the filtered page is empty to inform the accumulator about the group count
accumulator.addInput(groupIds, filtered, Optional.of(distinctMask));
}

Expand Down
11 changes: 10 additions & 1 deletion core/trino-main/src/main/java/io/trino/operator/aggregation/OrderedAccumulatorFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,16 @@ public void addInput(GroupByIdBlock groupIdsBlock, Page page, Optional<Block> ma
if (mask.isPresent()) {
page = filter(page, mask.orElseThrow());
}
pagesIndex.addPage(page);
if (page.getPositionCount() == 0) {
Copy link
Copy Markdown
Member

@sopel39 sopel39 Aug 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the same seems to be required for DistinctAccumulator

Copy link
Copy Markdown
Member Author

@dain dain Aug 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I looked at DistinctAccumulator also, and missed the place where it skips input

Copy link
Copy Markdown
Member Author

@dain dain Aug 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing distinct code is correct, but I added a test to verify that we don't introduce this bug in that code (also, I verified the test catches the bug if we change the code). I also added a comment to the distinct code to remind people about the issue

// page was entirely filtered out, but we need to inform the accumulator of the new group count
accumulator.addInput(
new GroupByIdBlock(groupCount, page.getBlock(page.getChannelCount() - 1)),
page.getColumns(argumentChannels),
Optional.empty());
}
else {
pagesIndex.addPage(page);
}
}

@Override
Expand Down
27 changes: 27 additions & 0 deletions core/trino-main/src/test/java/io/trino/sql/query/TestDistinctAggregations.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -297,4 +297,31 @@ public void testIpAddressDistinct()
" (IPADDRESS'2001:db8:0:0:1::1')) AS t (ipaddress_col)"))
.matches("VALUES IPADDRESS'2001:db8:0:0:1::1'");
}

@Test
public void testCompletelyFilteredGroup()
{
// This query filters out all values to a most groups, which results in an accumulator with no pages to sort.
// This can cause a failure if the ordering code does not inform the accumulator of the max row group.
assertThat(assertions.query("" +
"SELECT count(id) > 15000, sum(cardinality(v)) " +
"FROM ( " +
" SELECT " +
" id, " +
" array_agg(DISTINCT v) filter (WHERE v IS NOT NULL) AS v " +
" from ( " +
" ( " +
" SELECT 'filtered' AS id, cast('value' AS varchar) AS v " +
" FROM (VALUES 1, 2, 3, 4, 5, 6, 7, 8, 9, 10) " +
" ) " +
" UNION ALL " +
" ( " +
" SELECT cast(uuid() AS varchar) AS id, cast(null AS varchar) AS v " +
" FROM UNNEST(combinations(ARRAY[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], 5)) " +
" ) " +
" ) " +
" GROUP BY id " +
")"))
.matches("VALUES (TRUE, BIGINT '1')");
}
}
27 changes: 27 additions & 0 deletions core/trino-main/src/test/java/io/trino/sql/query/TestOrderedAggregation.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,4 +167,31 @@ public void testRepeatedSortItems()
assertThat(assertions.query("SELECT count(x ORDER BY y, y) FROM (VALUES ('a', 2)) t(x, y)"))
.matches("VALUES BIGINT '1'");
}

@Test
public void testCompletelyFilteredGroup()
{
// This query filters out all values to a most groups, which results in an accumulator with no pages to sort.
// This can cause a failure if the ordering code does not inform the accumulator of the max row group.
assertThat(assertions.query("" +
"SELECT count(id) > 15000, sum(cardinality(v)) " +
"FROM ( " +
" SELECT " +
" id, " +
" array_agg( v ORDER BY t DESC) filter (WHERE v IS NOT NULL) AS v " +
" FROM ( " +
" ( " +
" SELECT 'filtered' AS id, cast('value' AS varchar) AS v, 'sort' AS t " +
" FROM (VALUES 1, 2, 3, 4, 5, 6, 7, 8, 9, 10) " +
" ) " +
" UNION ALL " +
" ( " +
" SELECT cast(uuid() AS varchar) AS id, cast(null AS varchar) AS v, 'sort' AS t " +
" FROM UNNEST(combinations(ARRAY[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], 5)) " +
" ) " +
" ) " +
" GROUP BY id " +
")"))
.matches("VALUES (TRUE, BIGINT '10')");
}
}