Skip to content

Fix percentile bug #4539

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
qianheng-aws merged 3 commits into from
Oct 14, 2025
Merged

Fix percentile bug #4539

qianheng-aws merged 3 commits into from
Oct 14, 2025

Conversation

@xinyual
Copy link
Contributor

@xinyual xinyual commented Oct 14, 2025 •
edited
Loading

Description

Fix the mirror bug in percentile UDAF.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#4460

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@xinyual
fix percentile bug
b26eb24 
Signed-off-by: xinyual <[email protected]>
@xinyual
add IT
17f4ac0 
Signed-off-by: xinyual <[email protected]>
@yuancu yuancu added the bug Something isn't working label Oct 14, 2025
yuancu
yuancu reviewed Oct 14, 2025
View reviewed changes
integ-test/src/test/java/org/opensearch/sql/ppl/StatsCommandIT.java Outdated
JSONObject response =
executeQuery(
String.format(
"source=%s | stats percentile(balance, 50), min(balance)", TEST_INDEX_BANK));
Copy link
Collaborator

@yuancu yuancu Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this case cover the scenario where the argument of percentile is a decimal as that in the original issue does?

source=opensearch_dashboards_sample_data_logs 
| eval RamMB=CEIL(machine.ram/1000000.0) 
| stats bucket_nullable=false MIN(RamMB) as Min_RAM_MB, percentile(RamMB, 80) as P80_RAM_MB

Copy link
Contributor Author

@xinyual xinyual Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your notification. Already change the IT.

@xinyual
optimize it
4ac2b46 
Signed-off-by: xinyual <[email protected]>
yuancu
yuancu approved these changes Oct 14, 2025
View reviewed changes
Copy link
Collaborator

@yuancu yuancu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qianheng-aws qianheng-aws merged commit 8de0386 into opensearch-project:main Oct 14, 2025
32 of 33 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 14, 2025
@github-actions
Fix percentile bug (#4539)
ffe66fa 
* fix percentile bug

Signed-off-by: xinyual <[email protected]>

* add IT

Signed-off-by: xinyual <[email protected]>

* optimize it

Signed-off-by: xinyual <[email protected]>

---------

Signed-off-by: xinyual <[email protected]>
(cherry picked from commit 8de0386)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Oct 14, 2025
Merged
@xinyual xinyual mentioned this pull request Oct 15, 2025
Closed
4 tasks
LantaoJin added a commit that referenced this pull request Oct 15, 2025
@opensearch-trigger-bot @github-actions @LantaoJin
Fix percentile bug (#4539) (#4542)
c32f265 
* fix percentile bug



* add IT



* optimize it



---------


(cherry picked from commit 8de0386)

Signed-off-by: xinyual <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: Lantao Jin <[email protected]>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Lantao Jin <[email protected]>
ykmr1224 added a commit to ykmr1224/sql that referenced this pull request Oct 15, 2025
@ykmr1224
Merge main to map-remove:
6acef56 
commit cba8d02
Author: Tomoyuki MORITA <[email protected]>
Date:   Wed Oct 15 13:08:05 2025 -0700

    Add MAP_APPEND internal function to Calcite PPL (opensearch-project#4515)

    * Add MAP_APPEND internal function to Calcite PPL

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Minor fix

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Address comment

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Rebase and fix IT issue

    Signed-off-by: Tomoyuki Morita <[email protected]>

    ---------

    Signed-off-by: Tomoyuki Morita <[email protected]>

commit 3388dc7
Author: Lantao Jin <[email protected]>
Date:   Thu Oct 16 01:45:29 2025 +0800

    Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance (opensearch-project#4569)

    * Use _shard_doc as sort tiebreaker

    Signed-off-by: Lantao Jin <[email protected]>

    * _doc as a part of tie-breaker have better performance

    Signed-off-by: Lantao Jin <[email protected]>

    ---------

    Signed-off-by: Lantao Jin <[email protected]>

commit 5630119
Author: qianheng <[email protected]>
Date:   Wed Oct 15 16:40:41 2025 +0800

    Fix sort push down into agg after project already pushed (opensearch-project#4546)

    * Fix sort push down into agg

    Signed-off-by: Heng Qian <[email protected]>

    * Change some json files to yaml format

    Signed-off-by: Heng Qian <[email protected]>

    ---------

    Signed-off-by: Heng Qian <[email protected]>

commit 1e62fba
Author: Tomoyuki MORITA <[email protected]>
Date:   Tue Oct 14 17:20:38 2025 -0700

    Fix JsonExtractAllFunctionIT failure (opensearch-project#4556)

    Signed-off-by: Tomoyuki Morita <[email protected]>

commit 02ee33e
Author: Kai Huang <[email protected]>
Date:   Tue Oct 14 14:28:53 2025 -0700

    Add more examples to the `where` command doc (opensearch-project#4457)

    Co-authored-by: Manasvini B S <[email protected]>

commit 0b7e86c
Author: Jialiang Liang <[email protected]>
Date:   Tue Oct 14 10:46:01 2025 -0700

    [Enhancement] Error handling for illegal character usage in java regex named capture group (opensearch-project#4434)

    Co-authored-by: Simeon Widdis <[email protected]>

commit 9c97cfb
Author: Tomoyuki MORITA <[email protected]>
Date:   Tue Oct 14 08:36:43 2025 -0700

    Add JSON_EXTRACT_ALL internal function for Calcite PPL (opensearch-project#4489)

    * Add JSON_EXTRACT_ALL internal function for Calcite PPL

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Address comments

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Minor fix

    Signed-off-by: Tomoyuki Morita <[email protected]>

    ---------

    Signed-off-by: Tomoyuki Morita <[email protected]>

commit 89dbc31
Author: Lantao Jin <[email protected]>
Date:   Tue Oct 14 18:24:52 2025 +0800

    Check server status before starting Prometheus (opensearch-project#4537)

    * Check server status before starting Prometheus

    Signed-off-by: Lantao Jin <[email protected]>

    * Change to func call

    Signed-off-by: Lantao Jin <[email protected]>

    * Fix doc

    Signed-off-by: Lantao Jin <[email protected]>

    ---------

    Signed-off-by: Lantao Jin <[email protected]>

commit fe62472
Author: Lantao Jin <[email protected]>
Date:   Tue Oct 14 18:10:27 2025 +0800

    Update request builder after pushdown sort into agg buckets (opensearch-project#4541)

    Signed-off-by: Lantao Jin <[email protected]>

commit 42a415f
Author: qianheng <[email protected]>
Date:   Tue Oct 14 17:42:45 2025 +0800

    Including metadata fields type when doing agg/filter script push down (opensearch-project#4522)

    * Including metadata fields type when doing agg/filter script push down

    Signed-off-by: Heng Qian <[email protected]>

    * Fix IT

    Signed-off-by: Heng Qian <[email protected]>

    ---------

    Signed-off-by: Heng Qian <[email protected]>

commit 8de0386
Author: Xinyuan Lu <[email protected]>
Date:   Tue Oct 14 16:41:08 2025 +0800

    Fix percentile bug (opensearch-project#4539)

    * fix percentile bug

    Signed-off-by: xinyual <[email protected]>

    * add IT

    Signed-off-by: xinyual <[email protected]>

    * optimize it

    Signed-off-by: xinyual <[email protected]>

    ---------

    Signed-off-by: xinyual <[email protected]>

commit de2fdc8
Author: Lantao Jin <[email protected]>
Date:   Tue Oct 14 12:29:03 2025 +0800

    [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited (opensearch-project#4534)

    * [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited

    Signed-off-by: Lantao Jin <[email protected]>

    * fix doctest

    Signed-off-by: Lantao Jin <[email protected]>

    * Fix conflicts

    Signed-off-by: Lantao Jin <[email protected]>

    ---------

    Signed-off-by: Lantao Jin <[email protected]>

commit 977b7ab
Author: Simeon Widdis <[email protected]>
Date:   Mon Oct 13 20:23:10 2025 -0700

    Update stalled action (opensearch-project#4485)

commit fddbb70
Author: Lantao Jin <[email protected]>
Date:   Tue Oct 14 10:23:12 2025 +0800

    Add configurable sytem limitations for `subsearch` and `join` command (opensearch-project#4501)

    * Add configurable sytem limitations for subsearch and join command

    Signed-off-by: Lantao Jin <[email protected]>

    * Fix IT

    Signed-off-by: Lantao Jin <[email protected]>

    * typo

    Signed-off-by: Lantao Jin <[email protected]>

    * fix IT

    Signed-off-by: Lantao Jin <[email protected]>

    * remove rollback in doc

    Signed-off-by: Lantao Jin <[email protected]>

    * address comments

    Signed-off-by: Lantao Jin <[email protected]>

    * fix typo

    Signed-off-by: Lantao Jin <[email protected]>

    * Fix IT

    Signed-off-by: Lantao Jin <[email protected]>

    ---------

    Signed-off-by: Lantao Jin <[email protected]>

Signed-off-by: Tomoyuki Morita <[email protected]>
sandeshkr419 added a commit to sandeshkr419/sql that referenced this pull request Dec 3, 2025
@sandeshkr419 @github-actions
@Swiddis @manasvinibs @opensearch-ci-bot @noCharger @dai-chen @Mebsina @yuancu @opensearch-trigger-bot @ahkcs @penghuo @alexey-temnikov @RileyJergerAmazon @ykmr1224 @LantaoJin @songkant-aws @qianheng-aws @xinyual @RyanL1997 @peterzhuamazon @vinaykpud @expani @vamsimanohar @ritvibhatt @ishaoxy @mch2
Data fusion changes
6f7fd68 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Simeon Widdis <[email protected]>
Co-authored-by: Manasvini B S <[email protected]>
Co-authored-by: opensearch-ci-bot <[email protected]>
Co-authored-by: Louis Chu <[email protected]>
Co-authored-by: Chen Dai <[email protected]>
Co-authored-by: Mebsina <[email protected]>
Co-authored-by: Yuanchun Shen <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: Kai Huang <[email protected]>
Co-authored-by: Peng Huo <[email protected]>
Co-authored-by: Alexey Temnikov <[email protected]>
Co-authored-by: Riley Jerger <[email protected]>
Co-authored-by: Tomoyuki MORITA <[email protected]>
Co-authored-by: Lantao Jin <[email protected]>
Co-authored-by: Songkan Tang <[email protected]>
Co-authored-by: qianheng <[email protected]>
Co-authored-by: Simeon Widdis <[email protected]>
Co-authored-by: Xinyuan Lu <[email protected]>
Co-authored-by: Jialiang Liang <[email protected]>
Co-authored-by: Peter Zhu <[email protected]>
Co-authored-by: Vinay Krishna Pudyodu <[email protected]>
Co-authored-by: expani <[email protected]>
Co-authored-by: expani1729 <[email protected]>
Co-authored-by: Vamsi Manohar <[email protected]>
Co-authored-by: ritvibhatt <[email protected]>
Co-authored-by: Xinyu Hao <[email protected]>
Co-authored-by: Marc Handalian <[email protected]>
Co-authored-by: Marc Handalian <[email protected]>
Fix join type ambiguous issue when specify the join type with sql-like join criteria (opensearch-project#4474)
Fix issue 4441 (opensearch-project#4449)
Fix missing keywordsCanBeId (opensearch-project#4491)
Fix the bug of explicit makeNullLiteral for UDT fields (opensearch-project#4475)
Fix mapping after aggregation push down (opensearch-project#4500)
Fix percentile bug (opensearch-project#4539)
Fix JsonExtractAllFunctionIT failure (opensearch-project#4556)
Fix sort push down into agg after project already pushed (opensearch-project#4546)
Fix push down failure for min/max on derived field (opensearch-project#4572)
Fix compile issue in main (opensearch-project#4608)
Fix filter parsing failure on date fields with non-default format (opensearch-project#4616)
Fix bin nested fields issue (opensearch-project#4606)
Fix: Support Alias Fields in MIN, MAX, FIRST, LAST, and TAKE Aggregations (opensearch-project#4621)
fix rename issue (opensearch-project#4670)
Fixes for `Multisearch` and `Append` command (opensearch-project#4512)
Fix asc/desc keyword behavior for sort command (opensearch-project#4651)
Fix] Fix unexpected shift of extraction for `rex` with nested capture groups in named groups  (opensearch-project#4641)
Fix CVE-2025-48924 (opensearch-project#4665)
Fix sub-fields accessing of generated structs (opensearch-project#4683)
Fix] Incorrect Field Index Mapping in AVG to SUM/COUNT Conversion (opensearch-project#15)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yuancu yuancu yuancu approved these changes

@qianheng-aws qianheng-aws qianheng-aws approved these changes

@ps48 ps48 Awaiting requested review from ps48 ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen is a code owner

@YANG-DB YANG-DB Awaiting requested review from YANG-DB YANG-DB is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@vamsimanohar vamsimanohar Awaiting requested review from vamsimanohar vamsimanohar is a code owner

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@penghuo penghuo Awaiting requested review from penghuo penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey is a code owner

@acarbonetto acarbonetto Awaiting requested review from acarbonetto acarbonetto is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG is a code owner

@ykmr1224 ykmr1224 Awaiting requested review from ykmr1224 ykmr1224 is a code owner

@LantaoJin LantaoJin Awaiting requested review from LantaoJin LantaoJin is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger is a code owner

@RyanL1997 RyanL1997 Awaiting requested review from RyanL1997 RyanL1997 is a code owner

Assignees

No one assigned

Labels

backport 2.19-dev bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@xinyual @yuancu @qianheng-aws @LantaoJin