Skip to content

Conversation

@qianheng-aws
Copy link
Collaborator

@qianheng-aws qianheng-aws commented Aug 7, 2025

Description

Prevent aggregation push down when it has inner filter, also prevent push down agg on window function.

Related Issues

Resolves
#3996
#3999
#4009

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Heng Qian <[email protected]>
LantaoJin
LantaoJin previously approved these changes Aug 8, 2025
Signed-off-by: Heng Qian <[email protected]>
LantaoJin
LantaoJin previously approved these changes Aug 8, 2025
Signed-off-by: Heng Qian <[email protected]>
@Swiddis Swiddis merged commit 6e3329f into opensearch-project:main Aug 11, 2025
23 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Aug 11, 2025
* Prevent aggregation push down when it has inner filter

Signed-off-by: Heng Qian <[email protected]>

* Fix IT & Remove log

Signed-off-by: Heng Qian <[email protected]>

* Fix 4009

Signed-off-by: Heng Qian <[email protected]>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
(cherry picked from commit 6e3329f)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
ahkcs pushed a commit to ahkcs/sql that referenced this pull request Aug 11, 2025
…oject#4002)

* Prevent aggregation push down when it has inner filter

Signed-off-by: Heng Qian <[email protected]>

* Fix IT & Remove log

Signed-off-by: Heng Qian <[email protected]>

* Fix 4009

Signed-off-by: Heng Qian <[email protected]>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
ahkcs pushed a commit to ahkcs/sql that referenced this pull request Aug 11, 2025
…oject#4002)

* Prevent aggregation push down when it has inner filter

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Kai Huang <[email protected]>
@opensearch-trigger-bot
Copy link
Contributor

The backport to 3.2 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-3.2 3.2
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-3.2
# Create a new branch
git switch --create backport/backport-4002-to-3.2
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 6e3329faca3efd77a2f9746194664ccf8ea1df73
# Push it to GitHub
git push --set-upstream origin backport/backport-4002-to-3.2
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-3.2

Then, create a pull request where the base branch is 3.2 and the compare/head branch is backport/backport-4002-to-3.2.

@LantaoJin
Copy link
Member

Since this patch fixes a correctness issue, I suggest to backport it to 3.2.

qianheng-aws added a commit to qianheng-aws/sql that referenced this pull request Aug 12, 2025
…oject#4002)

* Prevent aggregation push down when it has inner filter

Signed-off-by: Heng Qian <[email protected]>

* Fix IT & Remove log

Signed-off-by: Heng Qian <[email protected]>

* Fix 4009

Signed-off-by: Heng Qian <[email protected]>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>

(cherry picked from commit 6e3329f)
Signed-off-by: Heng Qian <[email protected]>
LantaoJin pushed a commit that referenced this pull request Aug 13, 2025
…4002) (#4023)

* Prevent aggregation push down when it has inner filter (#4002)

* Prevent aggregation push down when it has inner filter

Signed-off-by: Heng Qian <[email protected]>

* Fix IT & Remove log

Signed-off-by: Heng Qian <[email protected]>

* Fix 4009

Signed-off-by: Heng Qian <[email protected]>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>

(cherry picked from commit 6e3329f)
Signed-off-by: Heng Qian <[email protected]>

* Remove code

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
RyanL1997 pushed a commit to RyanL1997/sql that referenced this pull request Aug 13, 2025
…oject#4002)

* Prevent aggregation push down when it has inner filter

Signed-off-by: Heng Qian <[email protected]>

* Fix IT & Remove log

Signed-off-by: Heng Qian <[email protected]>

* Fix 4009

Signed-off-by: Heng Qian <[email protected]>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
LantaoJin pushed a commit that referenced this pull request Aug 18, 2025
…ilter (#4022)

* Prevent aggregation push down when it has inner filter (#4002)

* Prevent aggregation push down when it has inner filter

Signed-off-by: Heng Qian <[email protected]>

* Fix IT & Remove log

Signed-off-by: Heng Qian <[email protected]>

* Fix 4009

Signed-off-by: Heng Qian <[email protected]>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
(cherry picked from commit 6e3329f)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Heng Qian <[email protected]>
@LantaoJin LantaoJin added the backport-manually Filed a PR to backport manually. label Aug 19, 2025
aalva500-prog pushed a commit to aalva500-prog/sql that referenced this pull request Aug 20, 2025
…oject#4002)

* Prevent aggregation push down when it has inner filter

Signed-off-by: Heng Qian <[email protected]>

* Fix IT & Remove log

Signed-off-by: Heng Qian <[email protected]>

* Fix 4009

Signed-off-by: Heng Qian <[email protected]>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>
dai-chen pushed a commit that referenced this pull request Aug 20, 2025
…cite) (#3970)

* Add support for space-separated fields in addition to comma-separated

Signed-off-by: Aaron Alvarez <[email protected]>

* Byte number should treated as Long in doc values (#3928)

Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Adding table as alias of the fields command

Signed-off-by: Aaron Alvarez <[email protected]>

* Fix create PIT permissions issue (#3921)

Signed-off-by: Vamsi Manohar <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Convert like function call to wildcard query for Calcite filter pushdown (#3915)

* Convert like function call to wildcard query for Calcite filter pushdown

Signed-off-by: Songkan Tang <[email protected]>

* Fix V2 expression like function bug and match its behavior in Calcite

Signed-off-by: Songkan Tang <[email protected]>

* Fix like default escape in Calcite

Signed-off-by: Songkan Tang <[email protected]>

* Fix tests

Signed-off-by: Songkan Tang <[email protected]>

* Fix spotless check

Signed-off-by: Songkan Tang <[email protected]>

* Address comments

Signed-off-by: Songkan Tang <[email protected]>

* Fix SQL IT correctness

Signed-off-by: Songkan Tang <[email protected]>

* Remove test log

Signed-off-by: Songkan Tang <[email protected]>

* Minor improve one CalciteLikeQueryIT

Signed-off-by: Songkan Tang <[email protected]>

---------

Signed-off-by: Songkan Tang <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Update commons-lang exclude rule to exclude it everywhere (#3932)

* Update commons-lang exclude rule to exclude it everywhere

Signed-off-by: Simeon Widdis <[email protected]>

* Undo removal in core

Signed-off-by: Simeon Widdis <[email protected]>

---------

Signed-off-by: Simeon Widdis <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Adding wildcard support to fields command in Calcite

Signed-off-by: Aaron Alvarez <[email protected]>

* Mixed delimiter support - Support both space and comma delimiters in the same command for table and fields

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding widlcard support to non-Calcite engine and updating documentation with new features

Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing formatting issues

Signed-off-by: Aaron Alvarez <[email protected]>

* Support function argument coercion with Calcite (#3914)

* Change the use of SqlTypeFamily.STRING to SqlTypeFamily.CHARACTER as the string family contains binary, which is not expected for most functions

Signed-off-by: Yuanchun Shen <[email protected]>

* Implement basic argument type coercion at RelNode level

Signed-off-by: Yuanchun Shen <[email protected]>

* Conform type checkers with their definition in documentation
- string as an input is removed if it is not in the document
- string as an input is kept if it is in the document, even if it can be implicitly cast
- use PPLOperandTypes as much as possible

Signed-off-by: Yuanchun Shen <[email protected]>

* Implement type widening for comparator functions

- Add COMPARATORS set to BuiltinFunctionName for identifying comparison operators
- Implement widenArguments method in CoercionUtils to find widest compatible type
- Apply type widening to comparator functions before applying type casting
- Add detailed JavaDoc to explain coercion methods

Signed-off-by: Yuanchun Shen <[email protected]>

* Update error messages of datetime functions with invalid args

Signed-off-by: Yuanchun Shen <[email protected]>

* Simplify datetime-string compare logic with implict coercion

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor resolve with coercion

Signed-off-by: Yuanchun Shen <[email protected]>

* Move down argument cast for reduce function

Signed-off-by: Yuanchun Shen <[email protected]>

* Merge comparators and their IP variants so that coercion works for IP comparison

- when not merging, ip comparing will also pass the type checker of Calcite's comparators

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor ip comparator to comparator

Signed-off-by: Yuanchun Shen <[email protected]>

* Revert "Refactor ip comparator to comparator"

This reverts commit c539056.

Signed-off-by: Yuanchun Shen <[email protected]>

* Revert "Merge comparators and their IP variants so that coercion works for IP comparison"

This reverts commit bd9f3bb.

Signed-off-by: Yuanchun Shen <[email protected]>

* Rule out ip from built-in comparator via its type checker

Signed-off-by: Yuanchun Shen <[email protected]>

* Restrict CompareIP's parameter type

Signed-off-by: Yuanchun Shen <[email protected]>

* Revert to previous implementation of CompareIpFunction to temporarily fix ip comparison pushdown problems (udt not correctly serialized; ip comparison is not converted to range query)

Signed-off-by: Yuanchun Shen <[email protected]>

* Test argument coercion explain

Signed-off-by: Yuanchun Shen <[email protected]>

* Fix error msg in CalcitePPLFunctionTypeTest

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Add missing command in index.rst (#3943)

Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Append limit operator for QUEERY_SIZE_LIMIT (#3940)

* Append limit operator for QUEERY_SIZE_LIMIT

Signed-off-by: Heng Qian <[email protected]>

* Add LogicalSystemLimit

Signed-off-by: Heng Qian <[email protected]>

* Revert part of #3880

Signed-off-by: Heng Qian <[email protected]>

* Fix IT after merging main

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Performing code cleaning and fixing tests

Signed-off-by: Aaron Alvarez <[email protected]>

* Performing code maintenance and adding more test cases

Signed-off-by: Aaron Alvarez <[email protected]>

* Doing some code cleaning and maintenance

Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing code and implementation logic

Signed-off-by: Aaron Alvarez <[email protected]>

* Add issue template specific for PPL commands and queries (#3962)

* Add issue template specific for PPL commands and queries

Signed-off-by: Anas Alkouz <[email protected]>

* Add section for Dataset/schema information, add reminders for the customers to remove any sensitive datas

Signed-off-by: Anas Alkouz <[email protected]>

---------

Signed-off-by: Anas Alkouz <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Increase the precision of sum return type (#3974)

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Disable a failed PPL query fallback to v2 by default (#3952)

* Disable a failed PPL query fallback to v2 by default

Signed-off-by: Lantao Jin <[email protected]>

* Workaround the permissionIT

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Update the maven snapshot publish endpoint and credential (#3806)

Co-authored-by: Sayali Gaikawad <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Add release notes for 3.2.0 (#3985)

Signed-off-by: opensearch-ci <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing documentation

Signed-off-by: Aaron Alvarez <[email protected]>

* Taking care of comments left by Tomo

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding full wildcard support functionality

Signed-off-by: Aaron Alvarez <[email protected]>

* Increment version to 3.2.0-SNAPSHOT (#3819)

Signed-off-by: opensearch-ci-bot <[email protected]>
Co-authored-by: opensearch-ci-bot <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Support `reverse` command with Calcite (#3867)

* Implement reverse

Signed-off-by: Selina Song <[email protected]>

* Add reverse integ tests and unit tests

Signed-off-by: Selina Song <[email protected]>

* Add reverse documentation

Signed-off-by: Selina Song <[email protected]>

* Modify reverse test and documentation

Signed-off-by: Selina Song <[email protected]>

* Fix limit pushdown bug when reverse comes before head

* Revert "Fix limit pushdown bug when reverse comes before head"

This reverts commit 087c936.

Signed-off-by: Selina Song <[email protected]>

* Fix grammar, naming, and test cases. Pushdown reverted will be in 2nd PR.

Signed-off-by: Selina Song <[email protected]>

* Fix reverse tests: update logical plans, format with Spotless

- Updated expected logical plans and Spark SQL in reverse tests
- Applied Spotless to fix formatting

Signed-off-by: Selina Song <[email protected]>

* Fix OS version in build 3.1.0

Signed-off-by: Selina Song <[email protected]>

* Add note on limitation to rst

Signed-off-by: Selina Song <[email protected]>

* Move explain IT to correct file, add Anonymizer test

Signed-off-by: Selina Song <[email protected]>

* Add reverse to index.rst

Signed-off-by: Selina Song <[email protected]>

---------

Signed-off-by: Selina Song <[email protected]>
Co-authored-by: Selina Song <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Pass JOIN_TIME_OUT value to keepalive (#3826)

* Fix JOIN_TIME_OUT

Signed-off-by: Kai Huang <[email protected]>

* fix style

Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* fix log-rethrow

Signed-off-by: Kai Huang <[email protected]>

* Restructure

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fix format

Signed-off-by: Kai Huang <[email protected]>

* remove unused PIT

Signed-off-by: Kai Huang <[email protected]>

* remove reflection, add hintConfig

Signed-off-by: Kai Huang <[email protected]>

* Added Unit test, restructured to use existing methods

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* add IT tests

Signed-off-by: Kai Huang <[email protected]>

* formatting

Signed-off-by: Kai Huang <[email protected]>

* Trim IT

Signed-off-by: Kai Huang <[email protected]>

* format fix

Signed-off-by: Kai Huang <[email protected]>

* trim IT

Signed-off-by: Kai Huang <[email protected]>

* formatting

Signed-off-by: Kai Huang <[email protected]>

* Update legacy/src/main/java/org/opensearch/sql/legacy/query/planner/physical/node/pointInTime/PointInTime.java

Co-authored-by: Louis Chu <[email protected]>
Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* update .gitignore

Signed-off-by: Kai Huang <[email protected]>

* deletion

Signed-off-by: Kai Huang <[email protected]>

* restore main PIT

Signed-off-by: Kai Huang <[email protected]>

---------

Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Co-authored-by: Louis Chu <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Changed seenFields to Hashset instead of LinkedHashSet

Signed-off-by: Aaron Alvarez <[email protected]>

* Creating a rule only for the fields/table commands to avoid any interference

Signed-off-by: Aaron Alvarez <[email protected]>

* This commit is in response of PR comments left by Tomo and Chen

Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing Integration test failure

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding anonymizer tests, wildcard unit tests, etc

Signed-off-by: Aaron Alvarez <[email protected]>

* Disabling Calcite for enhance fields features

Signed-off-by: Aaron Alvarez <[email protected]>

* Disabling automatic de-deduplication when Calcite is disabled

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding cross-cluster IT test

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding a dedicated Cross-cluster IT test file for Calcite

Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing formatting issues

Signed-off-by: Aaron Alvarez <[email protected]>

* Improving widlcard logic and exception message

Signed-off-by: Aaron Alvarez <[email protected]>

* Addressing comments left by Tomo regarding wildcard logic implementation

Signed-off-by: Aaron Alvarez <[email protected]>

* Empty commit

Signed-off-by: Aaron Alvarez <[email protected]>

* Add missing udfs in v3 (#3957)

* add math udfs

Signed-off-by: Xinyu Hao <[email protected]>

* fix decimal bug

Signed-off-by: Xinyu Hao <[email protected]>

* make general udf adapter

Signed-off-by: Xinyu Hao <[email protected]>

* add math IT

Signed-off-by: Xinyu Hao <[email protected]>

* fix

Signed-off-by: Xinyu Hao <[email protected]>

* add rst

Signed-off-by: Xinyu Hao <[email protected]>

* fix error

Signed-off-by: Xinyu Hao <[email protected]>

* change signum IT

Signed-off-by: Xinyu Hao <[email protected]>

* add javadoc

Signed-off-by: Xinyu Hao <[email protected]>

---------

Signed-off-by: Xinyu Hao <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* fix snapshot uploading (#4006)

* fix snapshot uploading

Signed-off-by: Kai Huang <[email protected]>

* Add comment

Signed-off-by: Kai Huang <[email protected]>

---------

Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fix DOUBLE to STRING cast rendering zero values in scientific notation (#3982)

* Fix casting double 0.0 to string

Signed-off-by: Yuanchun Shen <[email protected]>

* Fix float to string casting precision lost with custom FormatNumberFunction

This commit fixes float to string casting by replacing the use of SqlLibraryOperators.FORMAT_NUMBER
with a custom FormatNumberFunction implementation. The new implementation converts the number
to a BigDecimal before formatting to preserve precision and avoid issues like 6.2 becoming
6.199999809265137.

Signed-off-by: Yuanchun Shen <[email protected]>

* Simplify the implementation of fp number to string cast

Signed-off-by: Yuanchun Shen <[email protected]>

* Update implementation of NumberToStringFunction

Signed-off-by: Yuanchun Shen <[email protected]>

* Cast decimal with NUMBER_TO_STRING function

Signed-off-by: Yuanchun Shen <[email protected]>

* Test cast decimal

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Eliminate reliance on assert in Calcite for integration test (#4016)

* Move num-of-column check of in subquery ahead from RexSubQuery.java#L78 because assert is disabled in production

Signed-off-by: Yuanchun Shen <[email protected]>

* Increase script.context.filter.max_compilations_rate for SQLCorrectnessIT

Signed-off-by: Yuanchun Shen <[email protected]>

* Check script.disable_max_compilations_rate before setting context-specific compilations rate

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor: remove some methods in tests to upper level to reduce duplication

Signed-off-by: Yuanchun Shen <[email protected]>

* Replace plugin-level setting strings with private test-specific ones

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Prevent aggregation push down when it has inner filter (#4002)

* Prevent aggregation push down when it has inner filter

Signed-off-by: Heng Qian <[email protected]>

* Fix IT & Remove log

Signed-off-by: Heng Qian <[email protected]>

* Fix 4009

Signed-off-by: Heng Qian <[email protected]>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fix span on negative timestamp (#4017)

* Fix span on negative timestamp

Signed-off-by: Heng Qian <[email protected]>

* Fix span on negative timestamp

Signed-off-by: Heng Qian <[email protected]>

* typo

Signed-off-by: Heng Qian <[email protected]>

* Refine code

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Skip script encoding when run explain with  'extended' (#3930)

* No need to decode script when run explain command

Signed-off-by: Lantao Jin <[email protected]>

* address comment

Signed-off-by: Lantao Jin <[email protected]>

* Do not encoding when explain format is 'extended'

Signed-off-by: Lantao Jin <[email protected]>

* Rename the thread local var

Signed-off-by: Lantao Jin <[email protected]>

* Fix IT after merge main

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Implement type checking for aggregation functions with Calcite (#4024)

* Remove getTypeChecker from FunctionImp interface

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor registerExternalFunction to registerExternalOperator

Signed-off-by: Yuanchun Shen <[email protected]>

* Do not register GEOIP function if got incompatible client

Signed-off-by: Yuanchun Shen <[email protected]>

* Create scaffold for type checking of aggregation functions

Signed-off-by: Yuanchun Shen <[email protected]>

* Add type checkers for aggregation functions

Signed-off-by: Yuanchun Shen <[email protected]>

* Test type checking for aggregation functions

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Allow equal expression as a function argument (#4001)

* Remove named function arg from functions other than table functions

Signed-off-by: Yuanchun Shen <[email protected]>

* Test eval if function with equal as condition

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Push down IP comparison as range query with Calcite (#3959)

* Add reverse op for compare ip to support pushdown

Signed-off-by: Yuanchun Shen <[email protected]>

* Pushdown ip comparison

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor CompareIpFunction to use SqlKind directly

Signed-off-by: Yuanchun Shen <[email protected]>

* Simplify the overriding of reverse() for IP comparators

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* eval sum, avg implementation (#3986)

Signed-off-by: Vamsi Manohar <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fix PPL eval command string concatenation with + operator (#4020)

* eval command support

Signed-off-by: Kai Huang <[email protected]>

* improvment

Signed-off-by: Kai Huang <[email protected]>

* Refactor

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* Add IT

Signed-off-by: Kai Huang <[email protected]>

* remove redundant tests

Signed-off-by: Kai Huang <[email protected]>

---------

Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Support script push down on text field (#4010)

* Support script push down on text field

Signed-off-by: Heng Qian <[email protected]>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

* Add UT for struct type push down

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Enhance sort command in PPL (#3934)

* enhance sort command

Signed-off-by: Ritvi Bhatt <[email protected]>

* update failing tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix formatting

Signed-off-by: Ritvi Bhatt <[email protected]>

* add integ tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* update documentation

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix failing test

Signed-off-by: Ritvi Bhatt <[email protected]>

* update default and tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* update analyzer test

Signed-off-by: Ritvi Bhatt <[email protected]>

* update reverse sort direction

Signed-off-by: Ritvi Bhatt <[email protected]>

* update formatting

Signed-off-by: Ritvi Bhatt <[email protected]>

* update docs

Signed-off-by: Ritvi Bhatt <[email protected]>

* add javadoc

Signed-off-by: Ritvi Bhatt <[email protected]>

* add tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix failing tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix failing tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* update integ tests for query size limit change

Signed-off-by: Ritvi Bhatt <[email protected]>

* add explainit for desc and type cast

Signed-off-by: Ritvi Bhatt <[email protected]>

* add tests for desc

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix formatting

Signed-off-by: Ritvi Bhatt <[email protected]>

* make count optional

Signed-off-by: Ritvi Bhatt <[email protected]>

* add cross cluster tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* normalize count in AST node

Signed-off-by: Ritvi Bhatt <[email protected]>

* default null count to 0

Signed-off-by: Ritvi Bhatt <[email protected]>

* update logicalsort default constructor

Signed-off-by: Ritvi Bhatt <[email protected]>

---------

Signed-off-by: Ritvi Bhatt <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Add example for String concat in eval.rst (#4075)

* Add example for String concat in eval.rst

Signed-off-by: Kai Huang <[email protected]>

* mention calcite enabling

Signed-off-by: Kai Huang <[email protected]>

---------

Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Support pushdown dedup with Calcite (#3972)

* Support pushdown dedup with Calcite

Signed-off-by: Lantao Jin <[email protected]>

* Fix IT

Signed-off-by: Lantao Jin <[email protected]>

* Address comments

Signed-off-by: Lantao Jin <[email protected]>

* Fix flaky test

Signed-off-by: Lantao Jin <[email protected]>

* Address comment

Signed-off-by: Lantao Jin <[email protected]>

* delete useless codes

Signed-off-by: Lantao Jin <[email protected]>

* Add more ITs

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fix CI failure because of plan having changed (#4077)

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Empty commit

Signed-off-by: Aaron Alvarez <[email protected]>

* Empty commit

Signed-off-by: Aaron Alvarez <[email protected]>

---------

Signed-off-by: Aaron Alvarez <[email protected]>
Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Vamsi Manohar <[email protected]>
Signed-off-by: Songkan Tang <[email protected]>
Signed-off-by: Simeon Widdis <[email protected]>
Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Anas Alkouz <[email protected]>
Signed-off-by: opensearch-ci <[email protected]>
Signed-off-by: opensearch-ci-bot <[email protected]>
Signed-off-by: Selina Song <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Xinyu Hao <[email protected]>
Signed-off-by: Ritvi Bhatt <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>
Co-authored-by: Aaron Alvarez <[email protected]>
Co-authored-by: Lantao Jin <[email protected]>
Co-authored-by: Vamsi Manohar <[email protected]>
Co-authored-by: Songkan Tang <[email protected]>
Co-authored-by: Simeon Widdis <[email protected]>
Co-authored-by: Yuanchun Shen <[email protected]>
Co-authored-by: Peng Huo <[email protected]>
Co-authored-by: qianheng <[email protected]>
Co-authored-by: Anas Alkouz <[email protected]>
Co-authored-by: Zelin Hao <[email protected]>
Co-authored-by: Sayali Gaikawad <[email protected]>
Co-authored-by: opensearch-ci <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: opensearch-ci-bot <[email protected]>
Co-authored-by: Selina Song <[email protected]>
Co-authored-by: Selina Song <[email protected]>
Co-authored-by: Kai Huang <[email protected]>
Co-authored-by: Louis Chu <[email protected]>
Co-authored-by: Xinyu Hao <[email protected]>
Co-authored-by: ritvibhatt <[email protected]>
dai-chen pushed a commit to dai-chen/sql-1 that referenced this pull request Aug 20, 2025
…oject#4002)

* Prevent aggregation push down when it has inner filter

Signed-off-by: Heng Qian <[email protected]>

* Fix IT & Remove log

Signed-off-by: Heng Qian <[email protected]>

* Fix 4009

Signed-off-by: Heng Qian <[email protected]>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
aalva500-prog added a commit to aalva500-prog/sql that referenced this pull request Aug 21, 2025
…cite) (opensearch-project#3970)

* Add support for space-separated fields in addition to comma-separated

Signed-off-by: Aaron Alvarez <[email protected]>

* Byte number should treated as Long in doc values (opensearch-project#3928)

Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Adding table as alias of the fields command

Signed-off-by: Aaron Alvarez <[email protected]>

* Fix create PIT permissions issue (opensearch-project#3921)

Signed-off-by: Vamsi Manohar <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Convert like function call to wildcard query for Calcite filter pushdown (opensearch-project#3915)

* Convert like function call to wildcard query for Calcite filter pushdown

Signed-off-by: Songkan Tang <[email protected]>

* Fix V2 expression like function bug and match its behavior in Calcite

Signed-off-by: Songkan Tang <[email protected]>

* Fix like default escape in Calcite

Signed-off-by: Songkan Tang <[email protected]>

* Fix tests

Signed-off-by: Songkan Tang <[email protected]>

* Fix spotless check

Signed-off-by: Songkan Tang <[email protected]>

* Address comments

Signed-off-by: Songkan Tang <[email protected]>

* Fix SQL IT correctness

Signed-off-by: Songkan Tang <[email protected]>

* Remove test log

Signed-off-by: Songkan Tang <[email protected]>

* Minor improve one CalciteLikeQueryIT

Signed-off-by: Songkan Tang <[email protected]>

---------

Signed-off-by: Songkan Tang <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Update commons-lang exclude rule to exclude it everywhere (opensearch-project#3932)

* Update commons-lang exclude rule to exclude it everywhere

Signed-off-by: Simeon Widdis <[email protected]>

* Undo removal in core

Signed-off-by: Simeon Widdis <[email protected]>

---------

Signed-off-by: Simeon Widdis <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Adding wildcard support to fields command in Calcite

Signed-off-by: Aaron Alvarez <[email protected]>

* Mixed delimiter support - Support both space and comma delimiters in the same command for table and fields

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding widlcard support to non-Calcite engine and updating documentation with new features

Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing formatting issues

Signed-off-by: Aaron Alvarez <[email protected]>

* Support function argument coercion with Calcite (opensearch-project#3914)

* Change the use of SqlTypeFamily.STRING to SqlTypeFamily.CHARACTER as the string family contains binary, which is not expected for most functions

Signed-off-by: Yuanchun Shen <[email protected]>

* Implement basic argument type coercion at RelNode level

Signed-off-by: Yuanchun Shen <[email protected]>

* Conform type checkers with their definition in documentation
- string as an input is removed if it is not in the document
- string as an input is kept if it is in the document, even if it can be implicitly cast
- use PPLOperandTypes as much as possible

Signed-off-by: Yuanchun Shen <[email protected]>

* Implement type widening for comparator functions

- Add COMPARATORS set to BuiltinFunctionName for identifying comparison operators
- Implement widenArguments method in CoercionUtils to find widest compatible type
- Apply type widening to comparator functions before applying type casting
- Add detailed JavaDoc to explain coercion methods

Signed-off-by: Yuanchun Shen <[email protected]>

* Update error messages of datetime functions with invalid args

Signed-off-by: Yuanchun Shen <[email protected]>

* Simplify datetime-string compare logic with implict coercion

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor resolve with coercion

Signed-off-by: Yuanchun Shen <[email protected]>

* Move down argument cast for reduce function

Signed-off-by: Yuanchun Shen <[email protected]>

* Merge comparators and their IP variants so that coercion works for IP comparison

- when not merging, ip comparing will also pass the type checker of Calcite's comparators

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor ip comparator to comparator

Signed-off-by: Yuanchun Shen <[email protected]>

* Revert "Refactor ip comparator to comparator"

This reverts commit c539056.

Signed-off-by: Yuanchun Shen <[email protected]>

* Revert "Merge comparators and their IP variants so that coercion works for IP comparison"

This reverts commit bd9f3bb.

Signed-off-by: Yuanchun Shen <[email protected]>

* Rule out ip from built-in comparator via its type checker

Signed-off-by: Yuanchun Shen <[email protected]>

* Restrict CompareIP's parameter type

Signed-off-by: Yuanchun Shen <[email protected]>

* Revert to previous implementation of CompareIpFunction to temporarily fix ip comparison pushdown problems (udt not correctly serialized; ip comparison is not converted to range query)

Signed-off-by: Yuanchun Shen <[email protected]>

* Test argument coercion explain

Signed-off-by: Yuanchun Shen <[email protected]>

* Fix error msg in CalcitePPLFunctionTypeTest

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Add missing command in index.rst (opensearch-project#3943)

Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Append limit operator for QUEERY_SIZE_LIMIT (opensearch-project#3940)

* Append limit operator for QUEERY_SIZE_LIMIT

Signed-off-by: Heng Qian <[email protected]>

* Add LogicalSystemLimit

Signed-off-by: Heng Qian <[email protected]>

* Revert part of opensearch-project#3880

Signed-off-by: Heng Qian <[email protected]>

* Fix IT after merging main

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Performing code cleaning and fixing tests

Signed-off-by: Aaron Alvarez <[email protected]>

* Performing code maintenance and adding more test cases

Signed-off-by: Aaron Alvarez <[email protected]>

* Doing some code cleaning and maintenance

Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing code and implementation logic

Signed-off-by: Aaron Alvarez <[email protected]>

* Add issue template specific for PPL commands and queries (opensearch-project#3962)

* Add issue template specific for PPL commands and queries

Signed-off-by: Anas Alkouz <[email protected]>

* Add section for Dataset/schema information, add reminders for the customers to remove any sensitive datas

Signed-off-by: Anas Alkouz <[email protected]>

---------

Signed-off-by: Anas Alkouz <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Increase the precision of sum return type (opensearch-project#3974)

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Disable a failed PPL query fallback to v2 by default (opensearch-project#3952)

* Disable a failed PPL query fallback to v2 by default

Signed-off-by: Lantao Jin <[email protected]>

* Workaround the permissionIT

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Update the maven snapshot publish endpoint and credential (opensearch-project#3806)

Co-authored-by: Sayali Gaikawad <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Add release notes for 3.2.0 (opensearch-project#3985)

Signed-off-by: opensearch-ci <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing documentation

Signed-off-by: Aaron Alvarez <[email protected]>

* Taking care of comments left by Tomo

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding full wildcard support functionality

Signed-off-by: Aaron Alvarez <[email protected]>

* Increment version to 3.2.0-SNAPSHOT (opensearch-project#3819)

Signed-off-by: opensearch-ci-bot <[email protected]>
Co-authored-by: opensearch-ci-bot <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Support `reverse` command with Calcite (opensearch-project#3867)

* Implement reverse

Signed-off-by: Selina Song <[email protected]>

* Add reverse integ tests and unit tests

Signed-off-by: Selina Song <[email protected]>

* Add reverse documentation

Signed-off-by: Selina Song <[email protected]>

* Modify reverse test and documentation

Signed-off-by: Selina Song <[email protected]>

* Fix limit pushdown bug when reverse comes before head

* Revert "Fix limit pushdown bug when reverse comes before head"

This reverts commit 087c936.

Signed-off-by: Selina Song <[email protected]>

* Fix grammar, naming, and test cases. Pushdown reverted will be in 2nd PR.

Signed-off-by: Selina Song <[email protected]>

* Fix reverse tests: update logical plans, format with Spotless

- Updated expected logical plans and Spark SQL in reverse tests
- Applied Spotless to fix formatting

Signed-off-by: Selina Song <[email protected]>

* Fix OS version in build 3.1.0

Signed-off-by: Selina Song <[email protected]>

* Add note on limitation to rst

Signed-off-by: Selina Song <[email protected]>

* Move explain IT to correct file, add Anonymizer test

Signed-off-by: Selina Song <[email protected]>

* Add reverse to index.rst

Signed-off-by: Selina Song <[email protected]>

---------

Signed-off-by: Selina Song <[email protected]>
Co-authored-by: Selina Song <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Pass JOIN_TIME_OUT value to keepalive (opensearch-project#3826)

* Fix JOIN_TIME_OUT

Signed-off-by: Kai Huang <[email protected]>

* fix style

Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* fix log-rethrow

Signed-off-by: Kai Huang <[email protected]>

* Restructure

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fix format

Signed-off-by: Kai Huang <[email protected]>

* remove unused PIT

Signed-off-by: Kai Huang <[email protected]>

* remove reflection, add hintConfig

Signed-off-by: Kai Huang <[email protected]>

* Added Unit test, restructured to use existing methods

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* add IT tests

Signed-off-by: Kai Huang <[email protected]>

* formatting

Signed-off-by: Kai Huang <[email protected]>

* Trim IT

Signed-off-by: Kai Huang <[email protected]>

* format fix

Signed-off-by: Kai Huang <[email protected]>

* trim IT

Signed-off-by: Kai Huang <[email protected]>

* formatting

Signed-off-by: Kai Huang <[email protected]>

* Update legacy/src/main/java/org/opensearch/sql/legacy/query/planner/physical/node/pointInTime/PointInTime.java

Co-authored-by: Louis Chu <[email protected]>
Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* update .gitignore

Signed-off-by: Kai Huang <[email protected]>

* deletion

Signed-off-by: Kai Huang <[email protected]>

* restore main PIT

Signed-off-by: Kai Huang <[email protected]>

---------

Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Co-authored-by: Louis Chu <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Changed seenFields to Hashset instead of LinkedHashSet

Signed-off-by: Aaron Alvarez <[email protected]>

* Creating a rule only for the fields/table commands to avoid any interference

Signed-off-by: Aaron Alvarez <[email protected]>

* This commit is in response of PR comments left by Tomo and Chen

Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing Integration test failure

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding anonymizer tests, wildcard unit tests, etc

Signed-off-by: Aaron Alvarez <[email protected]>

* Disabling Calcite for enhance fields features

Signed-off-by: Aaron Alvarez <[email protected]>

* Disabling automatic de-deduplication when Calcite is disabled

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding cross-cluster IT test

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding a dedicated Cross-cluster IT test file for Calcite

Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing formatting issues

Signed-off-by: Aaron Alvarez <[email protected]>

* Improving widlcard logic and exception message

Signed-off-by: Aaron Alvarez <[email protected]>

* Addressing comments left by Tomo regarding wildcard logic implementation

Signed-off-by: Aaron Alvarez <[email protected]>

* Empty commit

Signed-off-by: Aaron Alvarez <[email protected]>

* Add missing udfs in v3 (opensearch-project#3957)

* add math udfs

Signed-off-by: Xinyu Hao <[email protected]>

* fix decimal bug

Signed-off-by: Xinyu Hao <[email protected]>

* make general udf adapter

Signed-off-by: Xinyu Hao <[email protected]>

* add math IT

Signed-off-by: Xinyu Hao <[email protected]>

* fix

Signed-off-by: Xinyu Hao <[email protected]>

* add rst

Signed-off-by: Xinyu Hao <[email protected]>

* fix error

Signed-off-by: Xinyu Hao <[email protected]>

* change signum IT

Signed-off-by: Xinyu Hao <[email protected]>

* add javadoc

Signed-off-by: Xinyu Hao <[email protected]>

---------

Signed-off-by: Xinyu Hao <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* fix snapshot uploading (opensearch-project#4006)

* fix snapshot uploading

Signed-off-by: Kai Huang <[email protected]>

* Add comment

Signed-off-by: Kai Huang <[email protected]>

---------

Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fix DOUBLE to STRING cast rendering zero values in scientific notation (opensearch-project#3982)

* Fix casting double 0.0 to string

Signed-off-by: Yuanchun Shen <[email protected]>

* Fix float to string casting precision lost with custom FormatNumberFunction

This commit fixes float to string casting by replacing the use of SqlLibraryOperators.FORMAT_NUMBER
with a custom FormatNumberFunction implementation. The new implementation converts the number
to a BigDecimal before formatting to preserve precision and avoid issues like 6.2 becoming
6.199999809265137.

Signed-off-by: Yuanchun Shen <[email protected]>

* Simplify the implementation of fp number to string cast

Signed-off-by: Yuanchun Shen <[email protected]>

* Update implementation of NumberToStringFunction

Signed-off-by: Yuanchun Shen <[email protected]>

* Cast decimal with NUMBER_TO_STRING function

Signed-off-by: Yuanchun Shen <[email protected]>

* Test cast decimal

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Eliminate reliance on assert in Calcite for integration test (opensearch-project#4016)

* Move num-of-column check of in subquery ahead from RexSubQuery.java#L78 because assert is disabled in production

Signed-off-by: Yuanchun Shen <[email protected]>

* Increase script.context.filter.max_compilations_rate for SQLCorrectnessIT

Signed-off-by: Yuanchun Shen <[email protected]>

* Check script.disable_max_compilations_rate before setting context-specific compilations rate

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor: remove some methods in tests to upper level to reduce duplication

Signed-off-by: Yuanchun Shen <[email protected]>

* Replace plugin-level setting strings with private test-specific ones

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Prevent aggregation push down when it has inner filter (opensearch-project#4002)

* Prevent aggregation push down when it has inner filter

Signed-off-by: Heng Qian <[email protected]>

* Fix IT & Remove log

Signed-off-by: Heng Qian <[email protected]>

* Fix 4009

Signed-off-by: Heng Qian <[email protected]>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fix span on negative timestamp (opensearch-project#4017)

* Fix span on negative timestamp

Signed-off-by: Heng Qian <[email protected]>

* Fix span on negative timestamp

Signed-off-by: Heng Qian <[email protected]>

* typo

Signed-off-by: Heng Qian <[email protected]>

* Refine code

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Skip script encoding when run explain with  'extended' (opensearch-project#3930)

* No need to decode script when run explain command

Signed-off-by: Lantao Jin <[email protected]>

* address comment

Signed-off-by: Lantao Jin <[email protected]>

* Do not encoding when explain format is 'extended'

Signed-off-by: Lantao Jin <[email protected]>

* Rename the thread local var

Signed-off-by: Lantao Jin <[email protected]>

* Fix IT after merge main

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Implement type checking for aggregation functions with Calcite (opensearch-project#4024)

* Remove getTypeChecker from FunctionImp interface

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor registerExternalFunction to registerExternalOperator

Signed-off-by: Yuanchun Shen <[email protected]>

* Do not register GEOIP function if got incompatible client

Signed-off-by: Yuanchun Shen <[email protected]>

* Create scaffold for type checking of aggregation functions

Signed-off-by: Yuanchun Shen <[email protected]>

* Add type checkers for aggregation functions

Signed-off-by: Yuanchun Shen <[email protected]>

* Test type checking for aggregation functions

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Allow equal expression as a function argument (opensearch-project#4001)

* Remove named function arg from functions other than table functions

Signed-off-by: Yuanchun Shen <[email protected]>

* Test eval if function with equal as condition

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Push down IP comparison as range query with Calcite (opensearch-project#3959)

* Add reverse op for compare ip to support pushdown

Signed-off-by: Yuanchun Shen <[email protected]>

* Pushdown ip comparison

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor CompareIpFunction to use SqlKind directly

Signed-off-by: Yuanchun Shen <[email protected]>

* Simplify the overriding of reverse() for IP comparators

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* eval sum, avg implementation (opensearch-project#3986)

Signed-off-by: Vamsi Manohar <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fix PPL eval command string concatenation with + operator (opensearch-project#4020)

* eval command support

Signed-off-by: Kai Huang <[email protected]>

* improvment

Signed-off-by: Kai Huang <[email protected]>

* Refactor

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* Add IT

Signed-off-by: Kai Huang <[email protected]>

* remove redundant tests

Signed-off-by: Kai Huang <[email protected]>

---------

Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Support script push down on text field (opensearch-project#4010)

* Support script push down on text field

Signed-off-by: Heng Qian <[email protected]>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

* Add UT for struct type push down

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Enhance sort command in PPL (opensearch-project#3934)

* enhance sort command

Signed-off-by: Ritvi Bhatt <[email protected]>

* update failing tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix formatting

Signed-off-by: Ritvi Bhatt <[email protected]>

* add integ tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* update documentation

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix failing test

Signed-off-by: Ritvi Bhatt <[email protected]>

* update default and tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* update analyzer test

Signed-off-by: Ritvi Bhatt <[email protected]>

* update reverse sort direction

Signed-off-by: Ritvi Bhatt <[email protected]>

* update formatting

Signed-off-by: Ritvi Bhatt <[email protected]>

* update docs

Signed-off-by: Ritvi Bhatt <[email protected]>

* add javadoc

Signed-off-by: Ritvi Bhatt <[email protected]>

* add tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix failing tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix failing tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* update integ tests for query size limit change

Signed-off-by: Ritvi Bhatt <[email protected]>

* add explainit for desc and type cast

Signed-off-by: Ritvi Bhatt <[email protected]>

* add tests for desc

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix formatting

Signed-off-by: Ritvi Bhatt <[email protected]>

* make count optional

Signed-off-by: Ritvi Bhatt <[email protected]>

* add cross cluster tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* normalize count in AST node

Signed-off-by: Ritvi Bhatt <[email protected]>

* default null count to 0

Signed-off-by: Ritvi Bhatt <[email protected]>

* update logicalsort default constructor

Signed-off-by: Ritvi Bhatt <[email protected]>

---------

Signed-off-by: Ritvi Bhatt <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Add example for String concat in eval.rst (opensearch-project#4075)

* Add example for String concat in eval.rst

Signed-off-by: Kai Huang <[email protected]>

* mention calcite enabling

Signed-off-by: Kai Huang <[email protected]>

---------

Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Support pushdown dedup with Calcite (opensearch-project#3972)

* Support pushdown dedup with Calcite

Signed-off-by: Lantao Jin <[email protected]>

* Fix IT

Signed-off-by: Lantao Jin <[email protected]>

* Address comments

Signed-off-by: Lantao Jin <[email protected]>

* Fix flaky test

Signed-off-by: Lantao Jin <[email protected]>

* Address comment

Signed-off-by: Lantao Jin <[email protected]>

* delete useless codes

Signed-off-by: Lantao Jin <[email protected]>

* Add more ITs

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fix CI failure because of plan having changed (opensearch-project#4077)

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Empty commit

Signed-off-by: Aaron Alvarez <[email protected]>

* Empty commit

Signed-off-by: Aaron Alvarez <[email protected]>

---------

Signed-off-by: Aaron Alvarez <[email protected]>
Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Vamsi Manohar <[email protected]>
Signed-off-by: Songkan Tang <[email protected]>
Signed-off-by: Simeon Widdis <[email protected]>
Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Anas Alkouz <[email protected]>
Signed-off-by: opensearch-ci <[email protected]>
Signed-off-by: opensearch-ci-bot <[email protected]>
Signed-off-by: Selina Song <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Xinyu Hao <[email protected]>
Signed-off-by: Ritvi Bhatt <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>
Co-authored-by: Aaron Alvarez <[email protected]>
Co-authored-by: Lantao Jin <[email protected]>
Co-authored-by: Vamsi Manohar <[email protected]>
Co-authored-by: Songkan Tang <[email protected]>
Co-authored-by: Simeon Widdis <[email protected]>
Co-authored-by: Yuanchun Shen <[email protected]>
Co-authored-by: Peng Huo <[email protected]>
Co-authored-by: qianheng <[email protected]>
Co-authored-by: Anas Alkouz <[email protected]>
Co-authored-by: Zelin Hao <[email protected]>
Co-authored-by: Sayali Gaikawad <[email protected]>
Co-authored-by: opensearch-ci <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: opensearch-ci-bot <[email protected]>
Co-authored-by: Selina Song <[email protected]>
Co-authored-by: Selina Song <[email protected]>
Co-authored-by: Kai Huang <[email protected]>
Co-authored-by: Louis Chu <[email protected]>
Co-authored-by: Xinyu Hao <[email protected]>
Co-authored-by: ritvibhatt <[email protected]>
(cherry picked from commit f16f1c0)
qianheng-aws added a commit that referenced this pull request Aug 22, 2025
…ection Features (Cal… (#4102)

* `fields` Command Enhancement - Advanced Field Selection Features (Calcite) (#3970)

* Add support for space-separated fields in addition to comma-separated

Signed-off-by: Aaron Alvarez <[email protected]>

* Byte number should treated as Long in doc values (#3928)

Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Adding table as alias of the fields command

Signed-off-by: Aaron Alvarez <[email protected]>

* Fix create PIT permissions issue (#3921)

Signed-off-by: Vamsi Manohar <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Convert like function call to wildcard query for Calcite filter pushdown (#3915)

* Convert like function call to wildcard query for Calcite filter pushdown

Signed-off-by: Songkan Tang <[email protected]>

* Fix V2 expression like function bug and match its behavior in Calcite

Signed-off-by: Songkan Tang <[email protected]>

* Fix like default escape in Calcite

Signed-off-by: Songkan Tang <[email protected]>

* Fix tests

Signed-off-by: Songkan Tang <[email protected]>

* Fix spotless check

Signed-off-by: Songkan Tang <[email protected]>

* Address comments

Signed-off-by: Songkan Tang <[email protected]>

* Fix SQL IT correctness

Signed-off-by: Songkan Tang <[email protected]>

* Remove test log

Signed-off-by: Songkan Tang <[email protected]>

* Minor improve one CalciteLikeQueryIT

Signed-off-by: Songkan Tang <[email protected]>

---------

Signed-off-by: Songkan Tang <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Update commons-lang exclude rule to exclude it everywhere (#3932)

* Update commons-lang exclude rule to exclude it everywhere

Signed-off-by: Simeon Widdis <[email protected]>

* Undo removal in core

Signed-off-by: Simeon Widdis <[email protected]>

---------

Signed-off-by: Simeon Widdis <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Adding wildcard support to fields command in Calcite

Signed-off-by: Aaron Alvarez <[email protected]>

* Mixed delimiter support - Support both space and comma delimiters in the same command for table and fields

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding widlcard support to non-Calcite engine and updating documentation with new features

Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing formatting issues

Signed-off-by: Aaron Alvarez <[email protected]>

* Support function argument coercion with Calcite (#3914)

* Change the use of SqlTypeFamily.STRING to SqlTypeFamily.CHARACTER as the string family contains binary, which is not expected for most functions

Signed-off-by: Yuanchun Shen <[email protected]>

* Implement basic argument type coercion at RelNode level

Signed-off-by: Yuanchun Shen <[email protected]>

* Conform type checkers with their definition in documentation
- string as an input is removed if it is not in the document
- string as an input is kept if it is in the document, even if it can be implicitly cast
- use PPLOperandTypes as much as possible

Signed-off-by: Yuanchun Shen <[email protected]>

* Implement type widening for comparator functions

- Add COMPARATORS set to BuiltinFunctionName for identifying comparison operators
- Implement widenArguments method in CoercionUtils to find widest compatible type
- Apply type widening to comparator functions before applying type casting
- Add detailed JavaDoc to explain coercion methods

Signed-off-by: Yuanchun Shen <[email protected]>

* Update error messages of datetime functions with invalid args

Signed-off-by: Yuanchun Shen <[email protected]>

* Simplify datetime-string compare logic with implict coercion

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor resolve with coercion

Signed-off-by: Yuanchun Shen <[email protected]>

* Move down argument cast for reduce function

Signed-off-by: Yuanchun Shen <[email protected]>

* Merge comparators and their IP variants so that coercion works for IP comparison

- when not merging, ip comparing will also pass the type checker of Calcite's comparators

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor ip comparator to comparator

Signed-off-by: Yuanchun Shen <[email protected]>

* Revert "Refactor ip comparator to comparator"

This reverts commit c539056.

Signed-off-by: Yuanchun Shen <[email protected]>

* Revert "Merge comparators and their IP variants so that coercion works for IP comparison"

This reverts commit bd9f3bb.

Signed-off-by: Yuanchun Shen <[email protected]>

* Rule out ip from built-in comparator via its type checker

Signed-off-by: Yuanchun Shen <[email protected]>

* Restrict CompareIP's parameter type

Signed-off-by: Yuanchun Shen <[email protected]>

* Revert to previous implementation of CompareIpFunction to temporarily fix ip comparison pushdown problems (udt not correctly serialized; ip comparison is not converted to range query)

Signed-off-by: Yuanchun Shen <[email protected]>

* Test argument coercion explain

Signed-off-by: Yuanchun Shen <[email protected]>

* Fix error msg in CalcitePPLFunctionTypeTest

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Add missing command in index.rst (#3943)

Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Append limit operator for QUEERY_SIZE_LIMIT (#3940)

* Append limit operator for QUEERY_SIZE_LIMIT

Signed-off-by: Heng Qian <[email protected]>

* Add LogicalSystemLimit

Signed-off-by: Heng Qian <[email protected]>

* Revert part of #3880

Signed-off-by: Heng Qian <[email protected]>

* Fix IT after merging main

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Performing code cleaning and fixing tests

Signed-off-by: Aaron Alvarez <[email protected]>

* Performing code maintenance and adding more test cases

Signed-off-by: Aaron Alvarez <[email protected]>

* Doing some code cleaning and maintenance

Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing code and implementation logic

Signed-off-by: Aaron Alvarez <[email protected]>

* Add issue template specific for PPL commands and queries (#3962)

* Add issue template specific for PPL commands and queries

Signed-off-by: Anas Alkouz <[email protected]>

* Add section for Dataset/schema information, add reminders for the customers to remove any sensitive datas

Signed-off-by: Anas Alkouz <[email protected]>

---------

Signed-off-by: Anas Alkouz <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Increase the precision of sum return type (#3974)

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Disable a failed PPL query fallback to v2 by default (#3952)

* Disable a failed PPL query fallback to v2 by default

Signed-off-by: Lantao Jin <[email protected]>

* Workaround the permissionIT

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Update the maven snapshot publish endpoint and credential (#3806)

Co-authored-by: Sayali Gaikawad <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Add release notes for 3.2.0 (#3985)

Signed-off-by: opensearch-ci <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing documentation

Signed-off-by: Aaron Alvarez <[email protected]>

* Taking care of comments left by Tomo

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding full wildcard support functionality

Signed-off-by: Aaron Alvarez <[email protected]>

* Increment version to 3.2.0-SNAPSHOT (#3819)

Signed-off-by: opensearch-ci-bot <[email protected]>
Co-authored-by: opensearch-ci-bot <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Support `reverse` command with Calcite (#3867)

* Implement reverse

Signed-off-by: Selina Song <[email protected]>

* Add reverse integ tests and unit tests

Signed-off-by: Selina Song <[email protected]>

* Add reverse documentation

Signed-off-by: Selina Song <[email protected]>

* Modify reverse test and documentation

Signed-off-by: Selina Song <[email protected]>

* Fix limit pushdown bug when reverse comes before head

* Revert "Fix limit pushdown bug when reverse comes before head"

This reverts commit 087c936.

Signed-off-by: Selina Song <[email protected]>

* Fix grammar, naming, and test cases. Pushdown reverted will be in 2nd PR.

Signed-off-by: Selina Song <[email protected]>

* Fix reverse tests: update logical plans, format with Spotless

- Updated expected logical plans and Spark SQL in reverse tests
- Applied Spotless to fix formatting

Signed-off-by: Selina Song <[email protected]>

* Fix OS version in build 3.1.0

Signed-off-by: Selina Song <[email protected]>

* Add note on limitation to rst

Signed-off-by: Selina Song <[email protected]>

* Move explain IT to correct file, add Anonymizer test

Signed-off-by: Selina Song <[email protected]>

* Add reverse to index.rst

Signed-off-by: Selina Song <[email protected]>

---------

Signed-off-by: Selina Song <[email protected]>
Co-authored-by: Selina Song <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Pass JOIN_TIME_OUT value to keepalive (#3826)

* Fix JOIN_TIME_OUT

Signed-off-by: Kai Huang <[email protected]>

* fix style

Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* fix log-rethrow

Signed-off-by: Kai Huang <[email protected]>

* Restructure

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fix format

Signed-off-by: Kai Huang <[email protected]>

* remove unused PIT

Signed-off-by: Kai Huang <[email protected]>

* remove reflection, add hintConfig

Signed-off-by: Kai Huang <[email protected]>

* Added Unit test, restructured to use existing methods

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* add IT tests

Signed-off-by: Kai Huang <[email protected]>

* formatting

Signed-off-by: Kai Huang <[email protected]>

* Trim IT

Signed-off-by: Kai Huang <[email protected]>

* format fix

Signed-off-by: Kai Huang <[email protected]>

* trim IT

Signed-off-by: Kai Huang <[email protected]>

* formatting

Signed-off-by: Kai Huang <[email protected]>

* Update legacy/src/main/java/org/opensearch/sql/legacy/query/planner/physical/node/pointInTime/PointInTime.java

Co-authored-by: Louis Chu <[email protected]>
Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* update .gitignore

Signed-off-by: Kai Huang <[email protected]>

* deletion

Signed-off-by: Kai Huang <[email protected]>

* restore main PIT

Signed-off-by: Kai Huang <[email protected]>

---------

Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Co-authored-by: Louis Chu <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Changed seenFields to Hashset instead of LinkedHashSet

Signed-off-by: Aaron Alvarez <[email protected]>

* Creating a rule only for the fields/table commands to avoid any interference

Signed-off-by: Aaron Alvarez <[email protected]>

* This commit is in response of PR comments left by Tomo and Chen

Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing Integration test failure

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding anonymizer tests, wildcard unit tests, etc

Signed-off-by: Aaron Alvarez <[email protected]>

* Disabling Calcite for enhance fields features

Signed-off-by: Aaron Alvarez <[email protected]>

* Disabling automatic de-deduplication when Calcite is disabled

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding cross-cluster IT test

Signed-off-by: Aaron Alvarez <[email protected]>

* Adding a dedicated Cross-cluster IT test file for Calcite

Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing formatting issues

Signed-off-by: Aaron Alvarez <[email protected]>

* Improving widlcard logic and exception message

Signed-off-by: Aaron Alvarez <[email protected]>

* Addressing comments left by Tomo regarding wildcard logic implementation

Signed-off-by: Aaron Alvarez <[email protected]>

* Empty commit

Signed-off-by: Aaron Alvarez <[email protected]>

* Add missing udfs in v3 (#3957)

* add math udfs

Signed-off-by: Xinyu Hao <[email protected]>

* fix decimal bug

Signed-off-by: Xinyu Hao <[email protected]>

* make general udf adapter

Signed-off-by: Xinyu Hao <[email protected]>

* add math IT

Signed-off-by: Xinyu Hao <[email protected]>

* fix

Signed-off-by: Xinyu Hao <[email protected]>

* add rst

Signed-off-by: Xinyu Hao <[email protected]>

* fix error

Signed-off-by: Xinyu Hao <[email protected]>

* change signum IT

Signed-off-by: Xinyu Hao <[email protected]>

* add javadoc

Signed-off-by: Xinyu Hao <[email protected]>

---------

Signed-off-by: Xinyu Hao <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* fix snapshot uploading (#4006)

* fix snapshot uploading

Signed-off-by: Kai Huang <[email protected]>

* Add comment

Signed-off-by: Kai Huang <[email protected]>

---------

Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fix DOUBLE to STRING cast rendering zero values in scientific notation (#3982)

* Fix casting double 0.0 to string

Signed-off-by: Yuanchun Shen <[email protected]>

* Fix float to string casting precision lost with custom FormatNumberFunction

This commit fixes float to string casting by replacing the use of SqlLibraryOperators.FORMAT_NUMBER
with a custom FormatNumberFunction implementation. The new implementation converts the number
to a BigDecimal before formatting to preserve precision and avoid issues like 6.2 becoming
6.199999809265137.

Signed-off-by: Yuanchun Shen <[email protected]>

* Simplify the implementation of fp number to string cast

Signed-off-by: Yuanchun Shen <[email protected]>

* Update implementation of NumberToStringFunction

Signed-off-by: Yuanchun Shen <[email protected]>

* Cast decimal with NUMBER_TO_STRING function

Signed-off-by: Yuanchun Shen <[email protected]>

* Test cast decimal

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Eliminate reliance on assert in Calcite for integration test (#4016)

* Move num-of-column check of in subquery ahead from RexSubQuery.java#L78 because assert is disabled in production

Signed-off-by: Yuanchun Shen <[email protected]>

* Increase script.context.filter.max_compilations_rate for SQLCorrectnessIT

Signed-off-by: Yuanchun Shen <[email protected]>

* Check script.disable_max_compilations_rate before setting context-specific compilations rate

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor: remove some methods in tests to upper level to reduce duplication

Signed-off-by: Yuanchun Shen <[email protected]>

* Replace plugin-level setting strings with private test-specific ones

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Prevent aggregation push down when it has inner filter (#4002)

* Prevent aggregation push down when it has inner filter

Signed-off-by: Heng Qian <[email protected]>

* Fix IT & Remove log

Signed-off-by: Heng Qian <[email protected]>

* Fix 4009

Signed-off-by: Heng Qian <[email protected]>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fix span on negative timestamp (#4017)

* Fix span on negative timestamp

Signed-off-by: Heng Qian <[email protected]>

* Fix span on negative timestamp

Signed-off-by: Heng Qian <[email protected]>

* typo

Signed-off-by: Heng Qian <[email protected]>

* Refine code

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Skip script encoding when run explain with  'extended' (#3930)

* No need to decode script when run explain command

Signed-off-by: Lantao Jin <[email protected]>

* address comment

Signed-off-by: Lantao Jin <[email protected]>

* Do not encoding when explain format is 'extended'

Signed-off-by: Lantao Jin <[email protected]>

* Rename the thread local var

Signed-off-by: Lantao Jin <[email protected]>

* Fix IT after merge main

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Implement type checking for aggregation functions with Calcite (#4024)

* Remove getTypeChecker from FunctionImp interface

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor registerExternalFunction to registerExternalOperator

Signed-off-by: Yuanchun Shen <[email protected]>

* Do not register GEOIP function if got incompatible client

Signed-off-by: Yuanchun Shen <[email protected]>

* Create scaffold for type checking of aggregation functions

Signed-off-by: Yuanchun Shen <[email protected]>

* Add type checkers for aggregation functions

Signed-off-by: Yuanchun Shen <[email protected]>

* Test type checking for aggregation functions

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Allow equal expression as a function argument (#4001)

* Remove named function arg from functions other than table functions

Signed-off-by: Yuanchun Shen <[email protected]>

* Test eval if function with equal as condition

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Push down IP comparison as range query with Calcite (#3959)

* Add reverse op for compare ip to support pushdown

Signed-off-by: Yuanchun Shen <[email protected]>

* Pushdown ip comparison

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor CompareIpFunction to use SqlKind directly

Signed-off-by: Yuanchun Shen <[email protected]>

* Simplify the overriding of reverse() for IP comparators

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* eval sum, avg implementation (#3986)

Signed-off-by: Vamsi Manohar <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fix PPL eval command string concatenation with + operator (#4020)

* eval command support

Signed-off-by: Kai Huang <[email protected]>

* improvment

Signed-off-by: Kai Huang <[email protected]>

* Refactor

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fix CI

Signed-off-by: Kai Huang <[email protected]>

* fixes

Signed-off-by: Kai Huang <[email protected]>

* fix

Signed-off-by: Kai Huang <[email protected]>

* Add IT

Signed-off-by: Kai Huang <[email protected]>

* remove redundant tests

Signed-off-by: Kai Huang <[email protected]>

---------

Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Support script push down on text field (#4010)

* Support script push down on text field

Signed-off-by: Heng Qian <[email protected]>

* Fix IT

Signed-off-by: Heng Qian <[email protected]>

* Add UT for struct type push down

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Enhance sort command in PPL (#3934)

* enhance sort command

Signed-off-by: Ritvi Bhatt <[email protected]>

* update failing tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix formatting

Signed-off-by: Ritvi Bhatt <[email protected]>

* add integ tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* update documentation

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix failing test

Signed-off-by: Ritvi Bhatt <[email protected]>

* update default and tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* update analyzer test

Signed-off-by: Ritvi Bhatt <[email protected]>

* update reverse sort direction

Signed-off-by: Ritvi Bhatt <[email protected]>

* update formatting

Signed-off-by: Ritvi Bhatt <[email protected]>

* update docs

Signed-off-by: Ritvi Bhatt <[email protected]>

* add javadoc

Signed-off-by: Ritvi Bhatt <[email protected]>

* add tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix failing tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix failing tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* update integ tests for query size limit change

Signed-off-by: Ritvi Bhatt <[email protected]>

* add explainit for desc and type cast

Signed-off-by: Ritvi Bhatt <[email protected]>

* add tests for desc

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix formatting

Signed-off-by: Ritvi Bhatt <[email protected]>

* make count optional

Signed-off-by: Ritvi Bhatt <[email protected]>

* add cross cluster tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* fix tests

Signed-off-by: Ritvi Bhatt <[email protected]>

* normalize count in AST node

Signed-off-by: Ritvi Bhatt <[email protected]>

* default null count to 0

Signed-off-by: Ritvi Bhatt <[email protected]>

* update logicalsort default constructor

Signed-off-by: Ritvi Bhatt <[email protected]>

---------

Signed-off-by: Ritvi Bhatt <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Add example for String concat in eval.rst (#4075)

* Add example for String concat in eval.rst

Signed-off-by: Kai Huang <[email protected]>

* mention calcite enabling

Signed-off-by: Kai Huang <[email protected]>

---------

Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Support pushdown dedup with Calcite (#3972)

* Support pushdown dedup with Calcite

Signed-off-by: Lantao Jin <[email protected]>

* Fix IT

Signed-off-by: Lantao Jin <[email protected]>

* Address comments

Signed-off-by: Lantao Jin <[email protected]>

* Fix flaky test

Signed-off-by: Lantao Jin <[email protected]>

* Address comment

Signed-off-by: Lantao Jin <[email protected]>

* delete useless codes

Signed-off-by: Lantao Jin <[email protected]>

* Add more ITs

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Fix CI failure because of plan having changed (#4077)

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>

* Empty commit

Signed-off-by: Aaron Alvarez <[email protected]>

* Empty commit

Signed-off-by: Aaron Alvarez <[email protected]>

---------

Signed-off-by: Aaron Alvarez <[email protected]>
Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Vamsi Manohar <[email protected]>
Signed-off-by: Songkan Tang <[email protected]>
Signed-off-by: Simeon Widdis <[email protected]>
Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Anas Alkouz <[email protected]>
Signed-off-by: opensearch-ci <[email protected]>
Signed-off-by: opensearch-ci-bot <[email protected]>
Signed-off-by: Selina Song <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Xinyu Hao <[email protected]>
Signed-off-by: Ritvi Bhatt <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>
Co-authored-by: Aaron Alvarez <[email protected]>
Co-authored-by: Lantao Jin <[email protected]>
Co-authored-by: Vamsi Manohar <[email protected]>
Co-authored-by: Songkan Tang <[email protected]>
Co-authored-by: Simeon Widdis <[email protected]>
Co-authored-by: Yuanchun Shen <[email protected]>
Co-authored-by: Peng Huo <[email protected]>
Co-authored-by: qianheng <[email protected]>
Co-authored-by: Anas Alkouz <[email protected]>
Co-authored-by: Zelin Hao <[email protected]>
Co-authored-by: Sayali Gaikawad <[email protected]>
Co-authored-by: opensearch-ci <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: opensearch-ci-bot <[email protected]>
Co-authored-by: Selina Song <[email protected]>
Co-authored-by: Selina Song <[email protected]>
Co-authored-by: Kai Huang <[email protected]>
Co-authored-by: Louis Chu <[email protected]>
Co-authored-by: Xinyu Hao <[email protected]>
Co-authored-by: ritvibhatt <[email protected]>
(cherry picked from commit f16f1c0)

* Fixing compatibility issues

Signed-off-by: Aaron Alvarez <[email protected]>

* Fixing JDK compatibility issues

Signed-off-by: Aaron Alvarez <[email protected]>

* fixing java 11 compatibility issues

Signed-off-by: Aaron Alvarez <[email protected]>

---------

Signed-off-by: Aaron Alvarez <[email protected]>
Signed-off-by: Lantao Jin <[email protected]>
Signed-off-by: Vamsi Manohar <[email protected]>
Signed-off-by: Songkan Tang <[email protected]>
Signed-off-by: Simeon Widdis <[email protected]>
Signed-off-by: Yuanchun Shen <[email protected]>
Signed-off-by: Peng Huo <[email protected]>
Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Anas Alkouz <[email protected]>
Signed-off-by: opensearch-ci <[email protected]>
Signed-off-by: opensearch-ci-bot <[email protected]>
Signed-off-by: Selina Song <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Xinyu Hao <[email protected]>
Signed-off-by: Ritvi Bhatt <[email protected]>
Signed-off-by: Aaron Alvarez <[email protected]>
Co-authored-by: Aaron Alvarez <[email protected]>
Co-authored-by: Lantao Jin <[email protected]>
Co-authored-by: Vamsi Manohar <[email protected]>
Co-authored-by: Songkan Tang <[email protected]>
Co-authored-by: Simeon Widdis <[email protected]>
Co-authored-by: Yuanchun Shen <[email protected]>
Co-authored-by: Peng Huo <[email protected]>
Co-authored-by: qianheng <[email protected]>
Co-authored-by: Anas Alkouz <[email protected]>
Co-authored-by: Zelin Hao <[email protected]>
Co-authored-by: Sayali Gaikawad <[email protected]>
Co-authored-by: opensearch-ci <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: opensearch-ci-bot <[email protected]>
Co-authored-by: Selina Song <[email protected]>
Co-authored-by: Selina Song <[email protected]>
Co-authored-by: Kai Huang <[email protected]>
Co-authored-by: Louis Chu <[email protected]>
Co-authored-by: Xinyu Hao <[email protected]>
Co-authored-by: ritvibhatt <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 2.19-dev backport 3.2 backport-failed backport-manually Filed a PR to backport manually. bug Something isn't working calcite calcite migration releated PPL Piped processing language

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants