Append limit operator for QUEERY_SIZE_LIMIT #3940

qianheng-aws · 2025-07-30T08:32:52Z

Description

Since we don't have plan to implement eval push down, we still need to append LIMIT operator explicitly for QUERY_SIZE_LIMIT, in order to cover the case of Project - Scan. See the source comment: #3880 (comment)

Related Issues

Resolves #3879

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Heng Qian <[email protected]>

LantaoJin · 2025-07-30T09:36:27Z

core/src/main/java/org/opensearch/sql/executor/QueryService.java

+   */
+  public RelNode optimize(RelNode plan, CalcitePlanContext context, boolean isExplain) {
+    if (isExplain) return plan;
+    context.relBuilder.limit(0, context.querySizeLimit);


can you add a case in ExplainIT?

It won't change the output plan of explain.

Add an IT to verify querySizeLimit pushdown.
Q, should we revert code of #3880?

We can reuse LogicalSystemLimit in explain. The physical plan is no diff. Then we can verify the querySizeLimit pushdown in explain IT.

Discussed with @LantaoJin offline, I will add a new LogicalSystemLimit operator to represent the limit operator added by our system. It will be used in many other places as well where we need to add a limitation. So it will contain a field of Type to distinguish them.

I will manually revert few lines related, although they are compatible. That Push down QUERY_SIZE_LIMIT #3880 also contains a bug fix related to limit push down.

Signed-off-by: Heng Qian <[email protected]>

…ePushDown

Signed-off-by: Heng Qian <[email protected]>

penghuo · 2025-07-31T18:35:49Z

integ-test/src/test/resources/expectedOutput/calcite/explain_filter_agg_push.json

  "calcite": {
-    "logical": "LogicalProject(avg_age=[$2], state=[$0], city=[$1])\n  LogicalAggregate(group=[{0, 1}], avg_age=[AVG($2)])\n    LogicalProject(state=[$7], city=[$5], age=[$8])\n      LogicalFilter(condition=[>($8, 30)])\n        CalciteLogicalIndexScan(table=[[OpenSearch, opensearch-sql_test_index_account]])\n",
-    "physical": "EnumerableCalc(expr#0..2=[{inputs}], avg_age=[$t2], state=[$t0], city=[$t1])\n  CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch-sql_test_index_account]], PushDownContext=[[PROJECT->[city, state, age], FILTER->>($2, 30), AGGREGATION->rel#:LogicalAggregate.NONE.[](input=RelSubset#,group={0, 1},avg_age=AVG($2))], OpenSearchRequestBuilder(sourceBuilder={\"from\":0,\"size\":0,\"timeout\":\"1m\",\"query\":{\"range\":{\"age\":{\"from\":30,\"to\":null,\"include_lower\":false,\"include_upper\":true,\"boost\":1.0}}},\"_source\":{\"includes\":[\"city\",\"state\",\"age\"],\"excludes\":[]},\"sort\":[{\"_doc\":{\"order\":\"asc\"}}],\"aggregations\":{\"composite_buckets\":{\"composite\":{\"size\":1000,\"sources\":[{\"state\":{\"terms\":{\"field\":\"state.keyword\",\"missing_bucket\":true,\"missing_order\":\"first\",\"order\":\"asc\"}}},{\"city\":{\"terms\":{\"field\":\"city.keyword\",\"missing_bucket\":true,\"missing_order\":\"first\",\"order\":\"asc\"}}}]},\"aggregations\":{\"avg_age\":{\"avg\":{\"field\":\"age\"}}}}}}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])\n"
+    "logical": "LogicalSystemLimit(fetch=[10000], type=[QUERY_SIZE_LIMIT])\n  LogicalProject(avg_age=[$2], state=[$0], city=[$1])\n    LogicalAggregate(group=[{0, 1}], avg_age=[AVG($2)])\n      LogicalProject(state=[$7], city=[$5], age=[$8])\n        LogicalFilter(condition=[>($8, 30)])\n          CalciteLogicalIndexScan(table=[[OpenSearch, opensearch-sql_test_index_account]])\n",


we should supportLimit pushdown through aggregation. track with #3961

opensearch-trigger-bot · 2025-08-01T02:28:10Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-3940-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5316c0a8e985a6fc895cf0714e6d813e5b72c653
# Push it to GitHub
git push --set-upstream origin backport/backport-3940-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-3940-to-2.19-dev.

* Append limit operator for QUEERY_SIZE_LIMIT Signed-off-by: Heng Qian <[email protected]> * Add LogicalSystemLimit Signed-off-by: Heng Qian <[email protected]> * Revert part of opensearch-project#3880 Signed-off-by: Heng Qian <[email protected]> * Fix IT after merging main Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> (cherry picked from commit 5316c0a)

* Append limit operator for QUEERY_SIZE_LIMIT * Add LogicalSystemLimit * Revert part of #3880 * Fix IT after merging main --------- (cherry picked from commit 5316c0a) Signed-off-by: Heng Qian <[email protected]>

* Append limit operator for QUEERY_SIZE_LIMIT Signed-off-by: Heng Qian <[email protected]> * Add LogicalSystemLimit Signed-off-by: Heng Qian <[email protected]> * Revert part of opensearch-project#3880 Signed-off-by: Heng Qian <[email protected]> * Fix IT after merging main Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]>

…cite) (#3970) * Add support for space-separated fields in addition to comma-separated Signed-off-by: Aaron Alvarez <[email protected]> * Byte number should treated as Long in doc values (#3928) Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Adding table as alias of the fields command Signed-off-by: Aaron Alvarez <[email protected]> * Fix create PIT permissions issue (#3921) Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Convert like function call to wildcard query for Calcite filter pushdown (#3915) * Convert like function call to wildcard query for Calcite filter pushdown Signed-off-by: Songkan Tang <[email protected]> * Fix V2 expression like function bug and match its behavior in Calcite Signed-off-by: Songkan Tang <[email protected]> * Fix like default escape in Calcite Signed-off-by: Songkan Tang <[email protected]> * Fix tests Signed-off-by: Songkan Tang <[email protected]> * Fix spotless check Signed-off-by: Songkan Tang <[email protected]> * Address comments Signed-off-by: Songkan Tang <[email protected]> * Fix SQL IT correctness Signed-off-by: Songkan Tang <[email protected]> * Remove test log Signed-off-by: Songkan Tang <[email protected]> * Minor improve one CalciteLikeQueryIT Signed-off-by: Songkan Tang <[email protected]> --------- Signed-off-by: Songkan Tang <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Update commons-lang exclude rule to exclude it everywhere (#3932) * Update commons-lang exclude rule to exclude it everywhere Signed-off-by: Simeon Widdis <[email protected]> * Undo removal in core Signed-off-by: Simeon Widdis <[email protected]> --------- Signed-off-by: Simeon Widdis <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Adding wildcard support to fields command in Calcite Signed-off-by: Aaron Alvarez <[email protected]> * Mixed delimiter support - Support both space and comma delimiters in the same command for table and fields Signed-off-by: Aaron Alvarez <[email protected]> * Adding widlcard support to non-Calcite engine and updating documentation with new features Signed-off-by: Aaron Alvarez <[email protected]> * Fixing formatting issues Signed-off-by: Aaron Alvarez <[email protected]> * Support function argument coercion with Calcite (#3914) * Change the use of SqlTypeFamily.STRING to SqlTypeFamily.CHARACTER as the string family contains binary, which is not expected for most functions Signed-off-by: Yuanchun Shen <[email protected]> * Implement basic argument type coercion at RelNode level Signed-off-by: Yuanchun Shen <[email protected]> * Conform type checkers with their definition in documentation - string as an input is removed if it is not in the document - string as an input is kept if it is in the document, even if it can be implicitly cast - use PPLOperandTypes as much as possible Signed-off-by: Yuanchun Shen <[email protected]> * Implement type widening for comparator functions - Add COMPARATORS set to BuiltinFunctionName for identifying comparison operators - Implement widenArguments method in CoercionUtils to find widest compatible type - Apply type widening to comparator functions before applying type casting - Add detailed JavaDoc to explain coercion methods Signed-off-by: Yuanchun Shen <[email protected]> * Update error messages of datetime functions with invalid args Signed-off-by: Yuanchun Shen <[email protected]> * Simplify datetime-string compare logic with implict coercion Signed-off-by: Yuanchun Shen <[email protected]> * Refactor resolve with coercion Signed-off-by: Yuanchun Shen <[email protected]> * Move down argument cast for reduce function Signed-off-by: Yuanchun Shen <[email protected]> * Merge comparators and their IP variants so that coercion works for IP comparison - when not merging, ip comparing will also pass the type checker of Calcite's comparators Signed-off-by: Yuanchun Shen <[email protected]> * Refactor ip comparator to comparator Signed-off-by: Yuanchun Shen <[email protected]> * Revert "Refactor ip comparator to comparator" This reverts commit c539056. Signed-off-by: Yuanchun Shen <[email protected]> * Revert "Merge comparators and their IP variants so that coercion works for IP comparison" This reverts commit bd9f3bb. Signed-off-by: Yuanchun Shen <[email protected]> * Rule out ip from built-in comparator via its type checker Signed-off-by: Yuanchun Shen <[email protected]> * Restrict CompareIP's parameter type Signed-off-by: Yuanchun Shen <[email protected]> * Revert to previous implementation of CompareIpFunction to temporarily fix ip comparison pushdown problems (udt not correctly serialized; ip comparison is not converted to range query) Signed-off-by: Yuanchun Shen <[email protected]> * Test argument coercion explain Signed-off-by: Yuanchun Shen <[email protected]> * Fix error msg in CalcitePPLFunctionTypeTest Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Add missing command in index.rst (#3943) Signed-off-by: Peng Huo <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Append limit operator for QUEERY_SIZE_LIMIT (#3940) * Append limit operator for QUEERY_SIZE_LIMIT Signed-off-by: Heng Qian <[email protected]> * Add LogicalSystemLimit Signed-off-by: Heng Qian <[email protected]> * Revert part of #3880 Signed-off-by: Heng Qian <[email protected]> * Fix IT after merging main Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Performing code cleaning and fixing tests Signed-off-by: Aaron Alvarez <[email protected]> * Performing code maintenance and adding more test cases Signed-off-by: Aaron Alvarez <[email protected]> * Doing some code cleaning and maintenance Signed-off-by: Aaron Alvarez <[email protected]> * Fixing code and implementation logic Signed-off-by: Aaron Alvarez <[email protected]> * Add issue template specific for PPL commands and queries (#3962) * Add issue template specific for PPL commands and queries Signed-off-by: Anas Alkouz <[email protected]> * Add section for Dataset/schema information, add reminders for the customers to remove any sensitive datas Signed-off-by: Anas Alkouz <[email protected]> --------- Signed-off-by: Anas Alkouz <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Increase the precision of sum return type (#3974) Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Disable a failed PPL query fallback to v2 by default (#3952) * Disable a failed PPL query fallback to v2 by default Signed-off-by: Lantao Jin <[email protected]> * Workaround the permissionIT Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Update the maven snapshot publish endpoint and credential (#3806) Co-authored-by: Sayali Gaikawad <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Add release notes for 3.2.0 (#3985) Signed-off-by: opensearch-ci <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fixing documentation Signed-off-by: Aaron Alvarez <[email protected]> * Taking care of comments left by Tomo Signed-off-by: Aaron Alvarez <[email protected]> * Adding full wildcard support functionality Signed-off-by: Aaron Alvarez <[email protected]> * Increment version to 3.2.0-SNAPSHOT (#3819) Signed-off-by: opensearch-ci-bot <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Support `reverse` command with Calcite (#3867) * Implement reverse Signed-off-by: Selina Song <[email protected]> * Add reverse integ tests and unit tests Signed-off-by: Selina Song <[email protected]> * Add reverse documentation Signed-off-by: Selina Song <[email protected]> * Modify reverse test and documentation Signed-off-by: Selina Song <[email protected]> * Fix limit pushdown bug when reverse comes before head * Revert "Fix limit pushdown bug when reverse comes before head" This reverts commit 087c936. Signed-off-by: Selina Song <[email protected]> * Fix grammar, naming, and test cases. Pushdown reverted will be in 2nd PR. Signed-off-by: Selina Song <[email protected]> * Fix reverse tests: update logical plans, format with Spotless - Updated expected logical plans and Spark SQL in reverse tests - Applied Spotless to fix formatting Signed-off-by: Selina Song <[email protected]> * Fix OS version in build 3.1.0 Signed-off-by: Selina Song <[email protected]> * Add note on limitation to rst Signed-off-by: Selina Song <[email protected]> * Move explain IT to correct file, add Anonymizer test Signed-off-by: Selina Song <[email protected]> * Add reverse to index.rst Signed-off-by: Selina Song <[email protected]> --------- Signed-off-by: Selina Song <[email protected]> Co-authored-by: Selina Song <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Pass JOIN_TIME_OUT value to keepalive (#3826) * Fix JOIN_TIME_OUT Signed-off-by: Kai Huang <[email protected]> * fix style Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * fix log-rethrow Signed-off-by: Kai Huang <[email protected]> * Restructure Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fix format Signed-off-by: Kai Huang <[email protected]> * remove unused PIT Signed-off-by: Kai Huang <[email protected]> * remove reflection, add hintConfig Signed-off-by: Kai Huang <[email protected]> * Added Unit test, restructured to use existing methods Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * add IT tests Signed-off-by: Kai Huang <[email protected]> * formatting Signed-off-by: Kai Huang <[email protected]> * Trim IT Signed-off-by: Kai Huang <[email protected]> * format fix Signed-off-by: Kai Huang <[email protected]> * trim IT Signed-off-by: Kai Huang <[email protected]> * formatting Signed-off-by: Kai Huang <[email protected]> * Update legacy/src/main/java/org/opensearch/sql/legacy/query/planner/physical/node/pointInTime/PointInTime.java Co-authored-by: Louis Chu <[email protected]> Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * update .gitignore Signed-off-by: Kai Huang <[email protected]> * deletion Signed-off-by: Kai Huang <[email protected]> * restore main PIT Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Kai Huang <[email protected]> Co-authored-by: Louis Chu <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Changed seenFields to Hashset instead of LinkedHashSet Signed-off-by: Aaron Alvarez <[email protected]> * Creating a rule only for the fields/table commands to avoid any interference Signed-off-by: Aaron Alvarez <[email protected]> * This commit is in response of PR comments left by Tomo and Chen Signed-off-by: Aaron Alvarez <[email protected]> * Fixing Integration test failure Signed-off-by: Aaron Alvarez <[email protected]> * Adding anonymizer tests, wildcard unit tests, etc Signed-off-by: Aaron Alvarez <[email protected]> * Disabling Calcite for enhance fields features Signed-off-by: Aaron Alvarez <[email protected]> * Disabling automatic de-deduplication when Calcite is disabled Signed-off-by: Aaron Alvarez <[email protected]> * Adding cross-cluster IT test Signed-off-by: Aaron Alvarez <[email protected]> * Adding a dedicated Cross-cluster IT test file for Calcite Signed-off-by: Aaron Alvarez <[email protected]> * Fixing formatting issues Signed-off-by: Aaron Alvarez <[email protected]> * Improving widlcard logic and exception message Signed-off-by: Aaron Alvarez <[email protected]> * Addressing comments left by Tomo regarding wildcard logic implementation Signed-off-by: Aaron Alvarez <[email protected]> * Empty commit Signed-off-by: Aaron Alvarez <[email protected]> * Add missing udfs in v3 (#3957) * add math udfs Signed-off-by: Xinyu Hao <[email protected]> * fix decimal bug Signed-off-by: Xinyu Hao <[email protected]> * make general udf adapter Signed-off-by: Xinyu Hao <[email protected]> * add math IT Signed-off-by: Xinyu Hao <[email protected]> * fix Signed-off-by: Xinyu Hao <[email protected]> * add rst Signed-off-by: Xinyu Hao <[email protected]> * fix error Signed-off-by: Xinyu Hao <[email protected]> * change signum IT Signed-off-by: Xinyu Hao <[email protected]> * add javadoc Signed-off-by: Xinyu Hao <[email protected]> --------- Signed-off-by: Xinyu Hao <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * fix snapshot uploading (#4006) * fix snapshot uploading Signed-off-by: Kai Huang <[email protected]> * Add comment Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fix DOUBLE to STRING cast rendering zero values in scientific notation (#3982) * Fix casting double 0.0 to string Signed-off-by: Yuanchun Shen <[email protected]> * Fix float to string casting precision lost with custom FormatNumberFunction This commit fixes float to string casting by replacing the use of SqlLibraryOperators.FORMAT_NUMBER with a custom FormatNumberFunction implementation. The new implementation converts the number to a BigDecimal before formatting to preserve precision and avoid issues like 6.2 becoming 6.199999809265137. Signed-off-by: Yuanchun Shen <[email protected]> * Simplify the implementation of fp number to string cast Signed-off-by: Yuanchun Shen <[email protected]> * Update implementation of NumberToStringFunction Signed-off-by: Yuanchun Shen <[email protected]> * Cast decimal with NUMBER_TO_STRING function Signed-off-by: Yuanchun Shen <[email protected]> * Test cast decimal Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Eliminate reliance on assert in Calcite for integration test (#4016) * Move num-of-column check of in subquery ahead from RexSubQuery.java#L78 because assert is disabled in production Signed-off-by: Yuanchun Shen <[email protected]> * Increase script.context.filter.max_compilations_rate for SQLCorrectnessIT Signed-off-by: Yuanchun Shen <[email protected]> * Check script.disable_max_compilations_rate before setting context-specific compilations rate Signed-off-by: Yuanchun Shen <[email protected]> * Refactor: remove some methods in tests to upper level to reduce duplication Signed-off-by: Yuanchun Shen <[email protected]> * Replace plugin-level setting strings with private test-specific ones Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Prevent aggregation push down when it has inner filter (#4002) * Prevent aggregation push down when it has inner filter Signed-off-by: Heng Qian <[email protected]> * Fix IT & Remove log Signed-off-by: Heng Qian <[email protected]> * Fix 4009 Signed-off-by: Heng Qian <[email protected]> * Fix IT Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fix span on negative timestamp (#4017) * Fix span on negative timestamp Signed-off-by: Heng Qian <[email protected]> * Fix span on negative timestamp Signed-off-by: Heng Qian <[email protected]> * typo Signed-off-by: Heng Qian <[email protected]> * Refine code Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Skip script encoding when run explain with 'extended' (#3930) * No need to decode script when run explain command Signed-off-by: Lantao Jin <[email protected]> * address comment Signed-off-by: Lantao Jin <[email protected]> * Do not encoding when explain format is 'extended' Signed-off-by: Lantao Jin <[email protected]> * Rename the thread local var Signed-off-by: Lantao Jin <[email protected]> * Fix IT after merge main Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Implement type checking for aggregation functions with Calcite (#4024) * Remove getTypeChecker from FunctionImp interface Signed-off-by: Yuanchun Shen <[email protected]> * Refactor registerExternalFunction to registerExternalOperator Signed-off-by: Yuanchun Shen <[email protected]> * Do not register GEOIP function if got incompatible client Signed-off-by: Yuanchun Shen <[email protected]> * Create scaffold for type checking of aggregation functions Signed-off-by: Yuanchun Shen <[email protected]> * Add type checkers for aggregation functions Signed-off-by: Yuanchun Shen <[email protected]> * Test type checking for aggregation functions Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Allow equal expression as a function argument (#4001) * Remove named function arg from functions other than table functions Signed-off-by: Yuanchun Shen <[email protected]> * Test eval if function with equal as condition Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Push down IP comparison as range query with Calcite (#3959) * Add reverse op for compare ip to support pushdown Signed-off-by: Yuanchun Shen <[email protected]> * Pushdown ip comparison Signed-off-by: Yuanchun Shen <[email protected]> * Refactor CompareIpFunction to use SqlKind directly Signed-off-by: Yuanchun Shen <[email protected]> * Simplify the overriding of reverse() for IP comparators Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * eval sum, avg implementation (#3986) Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fix PPL eval command string concatenation with + operator (#4020) * eval command support Signed-off-by: Kai Huang <[email protected]> * improvment Signed-off-by: Kai Huang <[email protected]> * Refactor Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * Add IT Signed-off-by: Kai Huang <[email protected]> * remove redundant tests Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Support script push down on text field (#4010) * Support script push down on text field Signed-off-by: Heng Qian <[email protected]> * Fix IT Signed-off-by: Heng Qian <[email protected]> * Add UT for struct type push down Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Enhance sort command in PPL (#3934) * enhance sort command Signed-off-by: Ritvi Bhatt <[email protected]> * update failing tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix formatting Signed-off-by: Ritvi Bhatt <[email protected]> * add integ tests Signed-off-by: Ritvi Bhatt <[email protected]> * update documentation Signed-off-by: Ritvi Bhatt <[email protected]> * fix failing test Signed-off-by: Ritvi Bhatt <[email protected]> * update default and tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix tests Signed-off-by: Ritvi Bhatt <[email protected]> * update analyzer test Signed-off-by: Ritvi Bhatt <[email protected]> * update reverse sort direction Signed-off-by: Ritvi Bhatt <[email protected]> * update formatting Signed-off-by: Ritvi Bhatt <[email protected]> * update docs Signed-off-by: Ritvi Bhatt <[email protected]> * add javadoc Signed-off-by: Ritvi Bhatt <[email protected]> * add tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix failing tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix failing tests Signed-off-by: Ritvi Bhatt <[email protected]> * update integ tests for query size limit change Signed-off-by: Ritvi Bhatt <[email protected]> * add explainit for desc and type cast Signed-off-by: Ritvi Bhatt <[email protected]> * add tests for desc Signed-off-by: Ritvi Bhatt <[email protected]> * fix formatting Signed-off-by: Ritvi Bhatt <[email protected]> * make count optional Signed-off-by: Ritvi Bhatt <[email protected]> * add cross cluster tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix tests Signed-off-by: Ritvi Bhatt <[email protected]> * normalize count in AST node Signed-off-by: Ritvi Bhatt <[email protected]> * default null count to 0 Signed-off-by: Ritvi Bhatt <[email protected]> * update logicalsort default constructor Signed-off-by: Ritvi Bhatt <[email protected]> --------- Signed-off-by: Ritvi Bhatt <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Add example for String concat in eval.rst (#4075) * Add example for String concat in eval.rst Signed-off-by: Kai Huang <[email protected]> * mention calcite enabling Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Support pushdown dedup with Calcite (#3972) * Support pushdown dedup with Calcite Signed-off-by: Lantao Jin <[email protected]> * Fix IT Signed-off-by: Lantao Jin <[email protected]> * Address comments Signed-off-by: Lantao Jin <[email protected]> * Fix flaky test Signed-off-by: Lantao Jin <[email protected]> * Address comment Signed-off-by: Lantao Jin <[email protected]> * delete useless codes Signed-off-by: Lantao Jin <[email protected]> * Add more ITs Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fix CI failure because of plan having changed (#4077) Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Empty commit Signed-off-by: Aaron Alvarez <[email protected]> * Empty commit Signed-off-by: Aaron Alvarez <[email protected]> --------- Signed-off-by: Aaron Alvarez <[email protected]> Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Songkan Tang <[email protected]> Signed-off-by: Simeon Widdis <[email protected]> Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Peng Huo <[email protected]> Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Anas Alkouz <[email protected]> Signed-off-by: opensearch-ci <[email protected]> Signed-off-by: opensearch-ci-bot <[email protected]> Signed-off-by: Selina Song <[email protected]> Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Xinyu Hao <[email protected]> Signed-off-by: Ritvi Bhatt <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> Co-authored-by: Aaron Alvarez <[email protected]> Co-authored-by: Lantao Jin <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]> Co-authored-by: Songkan Tang <[email protected]> Co-authored-by: Simeon Widdis <[email protected]> Co-authored-by: Yuanchun Shen <[email protected]> Co-authored-by: Peng Huo <[email protected]> Co-authored-by: qianheng <[email protected]> Co-authored-by: Anas Alkouz <[email protected]> Co-authored-by: Zelin Hao <[email protected]> Co-authored-by: Sayali Gaikawad <[email protected]> Co-authored-by: opensearch-ci <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Selina Song <[email protected]> Co-authored-by: Selina Song <[email protected]> Co-authored-by: Kai Huang <[email protected]> Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Xinyu Hao <[email protected]> Co-authored-by: ritvibhatt <[email protected]>

…cite) (opensearch-project#3970) * Add support for space-separated fields in addition to comma-separated Signed-off-by: Aaron Alvarez <[email protected]> * Byte number should treated as Long in doc values (opensearch-project#3928) Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Adding table as alias of the fields command Signed-off-by: Aaron Alvarez <[email protected]> * Fix create PIT permissions issue (opensearch-project#3921) Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Convert like function call to wildcard query for Calcite filter pushdown (opensearch-project#3915) * Convert like function call to wildcard query for Calcite filter pushdown Signed-off-by: Songkan Tang <[email protected]> * Fix V2 expression like function bug and match its behavior in Calcite Signed-off-by: Songkan Tang <[email protected]> * Fix like default escape in Calcite Signed-off-by: Songkan Tang <[email protected]> * Fix tests Signed-off-by: Songkan Tang <[email protected]> * Fix spotless check Signed-off-by: Songkan Tang <[email protected]> * Address comments Signed-off-by: Songkan Tang <[email protected]> * Fix SQL IT correctness Signed-off-by: Songkan Tang <[email protected]> * Remove test log Signed-off-by: Songkan Tang <[email protected]> * Minor improve one CalciteLikeQueryIT Signed-off-by: Songkan Tang <[email protected]> --------- Signed-off-by: Songkan Tang <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Update commons-lang exclude rule to exclude it everywhere (opensearch-project#3932) * Update commons-lang exclude rule to exclude it everywhere Signed-off-by: Simeon Widdis <[email protected]> * Undo removal in core Signed-off-by: Simeon Widdis <[email protected]> --------- Signed-off-by: Simeon Widdis <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Adding wildcard support to fields command in Calcite Signed-off-by: Aaron Alvarez <[email protected]> * Mixed delimiter support - Support both space and comma delimiters in the same command for table and fields Signed-off-by: Aaron Alvarez <[email protected]> * Adding widlcard support to non-Calcite engine and updating documentation with new features Signed-off-by: Aaron Alvarez <[email protected]> * Fixing formatting issues Signed-off-by: Aaron Alvarez <[email protected]> * Support function argument coercion with Calcite (opensearch-project#3914) * Change the use of SqlTypeFamily.STRING to SqlTypeFamily.CHARACTER as the string family contains binary, which is not expected for most functions Signed-off-by: Yuanchun Shen <[email protected]> * Implement basic argument type coercion at RelNode level Signed-off-by: Yuanchun Shen <[email protected]> * Conform type checkers with their definition in documentation - string as an input is removed if it is not in the document - string as an input is kept if it is in the document, even if it can be implicitly cast - use PPLOperandTypes as much as possible Signed-off-by: Yuanchun Shen <[email protected]> * Implement type widening for comparator functions - Add COMPARATORS set to BuiltinFunctionName for identifying comparison operators - Implement widenArguments method in CoercionUtils to find widest compatible type - Apply type widening to comparator functions before applying type casting - Add detailed JavaDoc to explain coercion methods Signed-off-by: Yuanchun Shen <[email protected]> * Update error messages of datetime functions with invalid args Signed-off-by: Yuanchun Shen <[email protected]> * Simplify datetime-string compare logic with implict coercion Signed-off-by: Yuanchun Shen <[email protected]> * Refactor resolve with coercion Signed-off-by: Yuanchun Shen <[email protected]> * Move down argument cast for reduce function Signed-off-by: Yuanchun Shen <[email protected]> * Merge comparators and their IP variants so that coercion works for IP comparison - when not merging, ip comparing will also pass the type checker of Calcite's comparators Signed-off-by: Yuanchun Shen <[email protected]> * Refactor ip comparator to comparator Signed-off-by: Yuanchun Shen <[email protected]> * Revert "Refactor ip comparator to comparator" This reverts commit c539056. Signed-off-by: Yuanchun Shen <[email protected]> * Revert "Merge comparators and their IP variants so that coercion works for IP comparison" This reverts commit bd9f3bb. Signed-off-by: Yuanchun Shen <[email protected]> * Rule out ip from built-in comparator via its type checker Signed-off-by: Yuanchun Shen <[email protected]> * Restrict CompareIP's parameter type Signed-off-by: Yuanchun Shen <[email protected]> * Revert to previous implementation of CompareIpFunction to temporarily fix ip comparison pushdown problems (udt not correctly serialized; ip comparison is not converted to range query) Signed-off-by: Yuanchun Shen <[email protected]> * Test argument coercion explain Signed-off-by: Yuanchun Shen <[email protected]> * Fix error msg in CalcitePPLFunctionTypeTest Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Add missing command in index.rst (opensearch-project#3943) Signed-off-by: Peng Huo <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Append limit operator for QUEERY_SIZE_LIMIT (opensearch-project#3940) * Append limit operator for QUEERY_SIZE_LIMIT Signed-off-by: Heng Qian <[email protected]> * Add LogicalSystemLimit Signed-off-by: Heng Qian <[email protected]> * Revert part of opensearch-project#3880 Signed-off-by: Heng Qian <[email protected]> * Fix IT after merging main Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Performing code cleaning and fixing tests Signed-off-by: Aaron Alvarez <[email protected]> * Performing code maintenance and adding more test cases Signed-off-by: Aaron Alvarez <[email protected]> * Doing some code cleaning and maintenance Signed-off-by: Aaron Alvarez <[email protected]> * Fixing code and implementation logic Signed-off-by: Aaron Alvarez <[email protected]> * Add issue template specific for PPL commands and queries (opensearch-project#3962) * Add issue template specific for PPL commands and queries Signed-off-by: Anas Alkouz <[email protected]> * Add section for Dataset/schema information, add reminders for the customers to remove any sensitive datas Signed-off-by: Anas Alkouz <[email protected]> --------- Signed-off-by: Anas Alkouz <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Increase the precision of sum return type (opensearch-project#3974) Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Disable a failed PPL query fallback to v2 by default (opensearch-project#3952) * Disable a failed PPL query fallback to v2 by default Signed-off-by: Lantao Jin <[email protected]> * Workaround the permissionIT Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Update the maven snapshot publish endpoint and credential (opensearch-project#3806) Co-authored-by: Sayali Gaikawad <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Add release notes for 3.2.0 (opensearch-project#3985) Signed-off-by: opensearch-ci <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fixing documentation Signed-off-by: Aaron Alvarez <[email protected]> * Taking care of comments left by Tomo Signed-off-by: Aaron Alvarez <[email protected]> * Adding full wildcard support functionality Signed-off-by: Aaron Alvarez <[email protected]> * Increment version to 3.2.0-SNAPSHOT (opensearch-project#3819) Signed-off-by: opensearch-ci-bot <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Support `reverse` command with Calcite (opensearch-project#3867) * Implement reverse Signed-off-by: Selina Song <[email protected]> * Add reverse integ tests and unit tests Signed-off-by: Selina Song <[email protected]> * Add reverse documentation Signed-off-by: Selina Song <[email protected]> * Modify reverse test and documentation Signed-off-by: Selina Song <[email protected]> * Fix limit pushdown bug when reverse comes before head * Revert "Fix limit pushdown bug when reverse comes before head" This reverts commit 087c936. Signed-off-by: Selina Song <[email protected]> * Fix grammar, naming, and test cases. Pushdown reverted will be in 2nd PR. Signed-off-by: Selina Song <[email protected]> * Fix reverse tests: update logical plans, format with Spotless - Updated expected logical plans and Spark SQL in reverse tests - Applied Spotless to fix formatting Signed-off-by: Selina Song <[email protected]> * Fix OS version in build 3.1.0 Signed-off-by: Selina Song <[email protected]> * Add note on limitation to rst Signed-off-by: Selina Song <[email protected]> * Move explain IT to correct file, add Anonymizer test Signed-off-by: Selina Song <[email protected]> * Add reverse to index.rst Signed-off-by: Selina Song <[email protected]> --------- Signed-off-by: Selina Song <[email protected]> Co-authored-by: Selina Song <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Pass JOIN_TIME_OUT value to keepalive (opensearch-project#3826) * Fix JOIN_TIME_OUT Signed-off-by: Kai Huang <[email protected]> * fix style Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * fix log-rethrow Signed-off-by: Kai Huang <[email protected]> * Restructure Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fix format Signed-off-by: Kai Huang <[email protected]> * remove unused PIT Signed-off-by: Kai Huang <[email protected]> * remove reflection, add hintConfig Signed-off-by: Kai Huang <[email protected]> * Added Unit test, restructured to use existing methods Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * add IT tests Signed-off-by: Kai Huang <[email protected]> * formatting Signed-off-by: Kai Huang <[email protected]> * Trim IT Signed-off-by: Kai Huang <[email protected]> * format fix Signed-off-by: Kai Huang <[email protected]> * trim IT Signed-off-by: Kai Huang <[email protected]> * formatting Signed-off-by: Kai Huang <[email protected]> * Update legacy/src/main/java/org/opensearch/sql/legacy/query/planner/physical/node/pointInTime/PointInTime.java Co-authored-by: Louis Chu <[email protected]> Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * update .gitignore Signed-off-by: Kai Huang <[email protected]> * deletion Signed-off-by: Kai Huang <[email protected]> * restore main PIT Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Kai Huang <[email protected]> Co-authored-by: Louis Chu <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Changed seenFields to Hashset instead of LinkedHashSet Signed-off-by: Aaron Alvarez <[email protected]> * Creating a rule only for the fields/table commands to avoid any interference Signed-off-by: Aaron Alvarez <[email protected]> * This commit is in response of PR comments left by Tomo and Chen Signed-off-by: Aaron Alvarez <[email protected]> * Fixing Integration test failure Signed-off-by: Aaron Alvarez <[email protected]> * Adding anonymizer tests, wildcard unit tests, etc Signed-off-by: Aaron Alvarez <[email protected]> * Disabling Calcite for enhance fields features Signed-off-by: Aaron Alvarez <[email protected]> * Disabling automatic de-deduplication when Calcite is disabled Signed-off-by: Aaron Alvarez <[email protected]> * Adding cross-cluster IT test Signed-off-by: Aaron Alvarez <[email protected]> * Adding a dedicated Cross-cluster IT test file for Calcite Signed-off-by: Aaron Alvarez <[email protected]> * Fixing formatting issues Signed-off-by: Aaron Alvarez <[email protected]> * Improving widlcard logic and exception message Signed-off-by: Aaron Alvarez <[email protected]> * Addressing comments left by Tomo regarding wildcard logic implementation Signed-off-by: Aaron Alvarez <[email protected]> * Empty commit Signed-off-by: Aaron Alvarez <[email protected]> * Add missing udfs in v3 (opensearch-project#3957) * add math udfs Signed-off-by: Xinyu Hao <[email protected]> * fix decimal bug Signed-off-by: Xinyu Hao <[email protected]> * make general udf adapter Signed-off-by: Xinyu Hao <[email protected]> * add math IT Signed-off-by: Xinyu Hao <[email protected]> * fix Signed-off-by: Xinyu Hao <[email protected]> * add rst Signed-off-by: Xinyu Hao <[email protected]> * fix error Signed-off-by: Xinyu Hao <[email protected]> * change signum IT Signed-off-by: Xinyu Hao <[email protected]> * add javadoc Signed-off-by: Xinyu Hao <[email protected]> --------- Signed-off-by: Xinyu Hao <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * fix snapshot uploading (opensearch-project#4006) * fix snapshot uploading Signed-off-by: Kai Huang <[email protected]> * Add comment Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fix DOUBLE to STRING cast rendering zero values in scientific notation (opensearch-project#3982) * Fix casting double 0.0 to string Signed-off-by: Yuanchun Shen <[email protected]> * Fix float to string casting precision lost with custom FormatNumberFunction This commit fixes float to string casting by replacing the use of SqlLibraryOperators.FORMAT_NUMBER with a custom FormatNumberFunction implementation. The new implementation converts the number to a BigDecimal before formatting to preserve precision and avoid issues like 6.2 becoming 6.199999809265137. Signed-off-by: Yuanchun Shen <[email protected]> * Simplify the implementation of fp number to string cast Signed-off-by: Yuanchun Shen <[email protected]> * Update implementation of NumberToStringFunction Signed-off-by: Yuanchun Shen <[email protected]> * Cast decimal with NUMBER_TO_STRING function Signed-off-by: Yuanchun Shen <[email protected]> * Test cast decimal Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Eliminate reliance on assert in Calcite for integration test (opensearch-project#4016) * Move num-of-column check of in subquery ahead from RexSubQuery.java#L78 because assert is disabled in production Signed-off-by: Yuanchun Shen <[email protected]> * Increase script.context.filter.max_compilations_rate for SQLCorrectnessIT Signed-off-by: Yuanchun Shen <[email protected]> * Check script.disable_max_compilations_rate before setting context-specific compilations rate Signed-off-by: Yuanchun Shen <[email protected]> * Refactor: remove some methods in tests to upper level to reduce duplication Signed-off-by: Yuanchun Shen <[email protected]> * Replace plugin-level setting strings with private test-specific ones Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Prevent aggregation push down when it has inner filter (opensearch-project#4002) * Prevent aggregation push down when it has inner filter Signed-off-by: Heng Qian <[email protected]> * Fix IT & Remove log Signed-off-by: Heng Qian <[email protected]> * Fix 4009 Signed-off-by: Heng Qian <[email protected]> * Fix IT Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fix span on negative timestamp (opensearch-project#4017) * Fix span on negative timestamp Signed-off-by: Heng Qian <[email protected]> * Fix span on negative timestamp Signed-off-by: Heng Qian <[email protected]> * typo Signed-off-by: Heng Qian <[email protected]> * Refine code Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Skip script encoding when run explain with 'extended' (opensearch-project#3930) * No need to decode script when run explain command Signed-off-by: Lantao Jin <[email protected]> * address comment Signed-off-by: Lantao Jin <[email protected]> * Do not encoding when explain format is 'extended' Signed-off-by: Lantao Jin <[email protected]> * Rename the thread local var Signed-off-by: Lantao Jin <[email protected]> * Fix IT after merge main Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Implement type checking for aggregation functions with Calcite (opensearch-project#4024) * Remove getTypeChecker from FunctionImp interface Signed-off-by: Yuanchun Shen <[email protected]> * Refactor registerExternalFunction to registerExternalOperator Signed-off-by: Yuanchun Shen <[email protected]> * Do not register GEOIP function if got incompatible client Signed-off-by: Yuanchun Shen <[email protected]> * Create scaffold for type checking of aggregation functions Signed-off-by: Yuanchun Shen <[email protected]> * Add type checkers for aggregation functions Signed-off-by: Yuanchun Shen <[email protected]> * Test type checking for aggregation functions Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Allow equal expression as a function argument (opensearch-project#4001) * Remove named function arg from functions other than table functions Signed-off-by: Yuanchun Shen <[email protected]> * Test eval if function with equal as condition Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Push down IP comparison as range query with Calcite (opensearch-project#3959) * Add reverse op for compare ip to support pushdown Signed-off-by: Yuanchun Shen <[email protected]> * Pushdown ip comparison Signed-off-by: Yuanchun Shen <[email protected]> * Refactor CompareIpFunction to use SqlKind directly Signed-off-by: Yuanchun Shen <[email protected]> * Simplify the overriding of reverse() for IP comparators Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * eval sum, avg implementation (opensearch-project#3986) Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fix PPL eval command string concatenation with + operator (opensearch-project#4020) * eval command support Signed-off-by: Kai Huang <[email protected]> * improvment Signed-off-by: Kai Huang <[email protected]> * Refactor Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * Add IT Signed-off-by: Kai Huang <[email protected]> * remove redundant tests Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Support script push down on text field (opensearch-project#4010) * Support script push down on text field Signed-off-by: Heng Qian <[email protected]> * Fix IT Signed-off-by: Heng Qian <[email protected]> * Add UT for struct type push down Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Enhance sort command in PPL (opensearch-project#3934) * enhance sort command Signed-off-by: Ritvi Bhatt <[email protected]> * update failing tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix formatting Signed-off-by: Ritvi Bhatt <[email protected]> * add integ tests Signed-off-by: Ritvi Bhatt <[email protected]> * update documentation Signed-off-by: Ritvi Bhatt <[email protected]> * fix failing test Signed-off-by: Ritvi Bhatt <[email protected]> * update default and tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix tests Signed-off-by: Ritvi Bhatt <[email protected]> * update analyzer test Signed-off-by: Ritvi Bhatt <[email protected]> * update reverse sort direction Signed-off-by: Ritvi Bhatt <[email protected]> * update formatting Signed-off-by: Ritvi Bhatt <[email protected]> * update docs Signed-off-by: Ritvi Bhatt <[email protected]> * add javadoc Signed-off-by: Ritvi Bhatt <[email protected]> * add tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix failing tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix failing tests Signed-off-by: Ritvi Bhatt <[email protected]> * update integ tests for query size limit change Signed-off-by: Ritvi Bhatt <[email protected]> * add explainit for desc and type cast Signed-off-by: Ritvi Bhatt <[email protected]> * add tests for desc Signed-off-by: Ritvi Bhatt <[email protected]> * fix formatting Signed-off-by: Ritvi Bhatt <[email protected]> * make count optional Signed-off-by: Ritvi Bhatt <[email protected]> * add cross cluster tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix tests Signed-off-by: Ritvi Bhatt <[email protected]> * normalize count in AST node Signed-off-by: Ritvi Bhatt <[email protected]> * default null count to 0 Signed-off-by: Ritvi Bhatt <[email protected]> * update logicalsort default constructor Signed-off-by: Ritvi Bhatt <[email protected]> --------- Signed-off-by: Ritvi Bhatt <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Add example for String concat in eval.rst (opensearch-project#4075) * Add example for String concat in eval.rst Signed-off-by: Kai Huang <[email protected]> * mention calcite enabling Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Support pushdown dedup with Calcite (opensearch-project#3972) * Support pushdown dedup with Calcite Signed-off-by: Lantao Jin <[email protected]> * Fix IT Signed-off-by: Lantao Jin <[email protected]> * Address comments Signed-off-by: Lantao Jin <[email protected]> * Fix flaky test Signed-off-by: Lantao Jin <[email protected]> * Address comment Signed-off-by: Lantao Jin <[email protected]> * delete useless codes Signed-off-by: Lantao Jin <[email protected]> * Add more ITs Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fix CI failure because of plan having changed (opensearch-project#4077) Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Empty commit Signed-off-by: Aaron Alvarez <[email protected]> * Empty commit Signed-off-by: Aaron Alvarez <[email protected]> --------- Signed-off-by: Aaron Alvarez <[email protected]> Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Songkan Tang <[email protected]> Signed-off-by: Simeon Widdis <[email protected]> Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Peng Huo <[email protected]> Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Anas Alkouz <[email protected]> Signed-off-by: opensearch-ci <[email protected]> Signed-off-by: opensearch-ci-bot <[email protected]> Signed-off-by: Selina Song <[email protected]> Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Xinyu Hao <[email protected]> Signed-off-by: Ritvi Bhatt <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> Co-authored-by: Aaron Alvarez <[email protected]> Co-authored-by: Lantao Jin <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]> Co-authored-by: Songkan Tang <[email protected]> Co-authored-by: Simeon Widdis <[email protected]> Co-authored-by: Yuanchun Shen <[email protected]> Co-authored-by: Peng Huo <[email protected]> Co-authored-by: qianheng <[email protected]> Co-authored-by: Anas Alkouz <[email protected]> Co-authored-by: Zelin Hao <[email protected]> Co-authored-by: Sayali Gaikawad <[email protected]> Co-authored-by: opensearch-ci <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Selina Song <[email protected]> Co-authored-by: Selina Song <[email protected]> Co-authored-by: Kai Huang <[email protected]> Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Xinyu Hao <[email protected]> Co-authored-by: ritvibhatt <[email protected]> (cherry picked from commit f16f1c0)

…ection Features (Cal… (#4102) * `fields` Command Enhancement - Advanced Field Selection Features (Calcite) (#3970) * Add support for space-separated fields in addition to comma-separated Signed-off-by: Aaron Alvarez <[email protected]> * Byte number should treated as Long in doc values (#3928) Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Adding table as alias of the fields command Signed-off-by: Aaron Alvarez <[email protected]> * Fix create PIT permissions issue (#3921) Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Convert like function call to wildcard query for Calcite filter pushdown (#3915) * Convert like function call to wildcard query for Calcite filter pushdown Signed-off-by: Songkan Tang <[email protected]> * Fix V2 expression like function bug and match its behavior in Calcite Signed-off-by: Songkan Tang <[email protected]> * Fix like default escape in Calcite Signed-off-by: Songkan Tang <[email protected]> * Fix tests Signed-off-by: Songkan Tang <[email protected]> * Fix spotless check Signed-off-by: Songkan Tang <[email protected]> * Address comments Signed-off-by: Songkan Tang <[email protected]> * Fix SQL IT correctness Signed-off-by: Songkan Tang <[email protected]> * Remove test log Signed-off-by: Songkan Tang <[email protected]> * Minor improve one CalciteLikeQueryIT Signed-off-by: Songkan Tang <[email protected]> --------- Signed-off-by: Songkan Tang <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Update commons-lang exclude rule to exclude it everywhere (#3932) * Update commons-lang exclude rule to exclude it everywhere Signed-off-by: Simeon Widdis <[email protected]> * Undo removal in core Signed-off-by: Simeon Widdis <[email protected]> --------- Signed-off-by: Simeon Widdis <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Adding wildcard support to fields command in Calcite Signed-off-by: Aaron Alvarez <[email protected]> * Mixed delimiter support - Support both space and comma delimiters in the same command for table and fields Signed-off-by: Aaron Alvarez <[email protected]> * Adding widlcard support to non-Calcite engine and updating documentation with new features Signed-off-by: Aaron Alvarez <[email protected]> * Fixing formatting issues Signed-off-by: Aaron Alvarez <[email protected]> * Support function argument coercion with Calcite (#3914) * Change the use of SqlTypeFamily.STRING to SqlTypeFamily.CHARACTER as the string family contains binary, which is not expected for most functions Signed-off-by: Yuanchun Shen <[email protected]> * Implement basic argument type coercion at RelNode level Signed-off-by: Yuanchun Shen <[email protected]> * Conform type checkers with their definition in documentation - string as an input is removed if it is not in the document - string as an input is kept if it is in the document, even if it can be implicitly cast - use PPLOperandTypes as much as possible Signed-off-by: Yuanchun Shen <[email protected]> * Implement type widening for comparator functions - Add COMPARATORS set to BuiltinFunctionName for identifying comparison operators - Implement widenArguments method in CoercionUtils to find widest compatible type - Apply type widening to comparator functions before applying type casting - Add detailed JavaDoc to explain coercion methods Signed-off-by: Yuanchun Shen <[email protected]> * Update error messages of datetime functions with invalid args Signed-off-by: Yuanchun Shen <[email protected]> * Simplify datetime-string compare logic with implict coercion Signed-off-by: Yuanchun Shen <[email protected]> * Refactor resolve with coercion Signed-off-by: Yuanchun Shen <[email protected]> * Move down argument cast for reduce function Signed-off-by: Yuanchun Shen <[email protected]> * Merge comparators and their IP variants so that coercion works for IP comparison - when not merging, ip comparing will also pass the type checker of Calcite's comparators Signed-off-by: Yuanchun Shen <[email protected]> * Refactor ip comparator to comparator Signed-off-by: Yuanchun Shen <[email protected]> * Revert "Refactor ip comparator to comparator" This reverts commit c539056. Signed-off-by: Yuanchun Shen <[email protected]> * Revert "Merge comparators and their IP variants so that coercion works for IP comparison" This reverts commit bd9f3bb. Signed-off-by: Yuanchun Shen <[email protected]> * Rule out ip from built-in comparator via its type checker Signed-off-by: Yuanchun Shen <[email protected]> * Restrict CompareIP's parameter type Signed-off-by: Yuanchun Shen <[email protected]> * Revert to previous implementation of CompareIpFunction to temporarily fix ip comparison pushdown problems (udt not correctly serialized; ip comparison is not converted to range query) Signed-off-by: Yuanchun Shen <[email protected]> * Test argument coercion explain Signed-off-by: Yuanchun Shen <[email protected]> * Fix error msg in CalcitePPLFunctionTypeTest Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Add missing command in index.rst (#3943) Signed-off-by: Peng Huo <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Append limit operator for QUEERY_SIZE_LIMIT (#3940) * Append limit operator for QUEERY_SIZE_LIMIT Signed-off-by: Heng Qian <[email protected]> * Add LogicalSystemLimit Signed-off-by: Heng Qian <[email protected]> * Revert part of #3880 Signed-off-by: Heng Qian <[email protected]> * Fix IT after merging main Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Performing code cleaning and fixing tests Signed-off-by: Aaron Alvarez <[email protected]> * Performing code maintenance and adding more test cases Signed-off-by: Aaron Alvarez <[email protected]> * Doing some code cleaning and maintenance Signed-off-by: Aaron Alvarez <[email protected]> * Fixing code and implementation logic Signed-off-by: Aaron Alvarez <[email protected]> * Add issue template specific for PPL commands and queries (#3962) * Add issue template specific for PPL commands and queries Signed-off-by: Anas Alkouz <[email protected]> * Add section for Dataset/schema information, add reminders for the customers to remove any sensitive datas Signed-off-by: Anas Alkouz <[email protected]> --------- Signed-off-by: Anas Alkouz <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Increase the precision of sum return type (#3974) Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Disable a failed PPL query fallback to v2 by default (#3952) * Disable a failed PPL query fallback to v2 by default Signed-off-by: Lantao Jin <[email protected]> * Workaround the permissionIT Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Update the maven snapshot publish endpoint and credential (#3806) Co-authored-by: Sayali Gaikawad <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Add release notes for 3.2.0 (#3985) Signed-off-by: opensearch-ci <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fixing documentation Signed-off-by: Aaron Alvarez <[email protected]> * Taking care of comments left by Tomo Signed-off-by: Aaron Alvarez <[email protected]> * Adding full wildcard support functionality Signed-off-by: Aaron Alvarez <[email protected]> * Increment version to 3.2.0-SNAPSHOT (#3819) Signed-off-by: opensearch-ci-bot <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Support `reverse` command with Calcite (#3867) * Implement reverse Signed-off-by: Selina Song <[email protected]> * Add reverse integ tests and unit tests Signed-off-by: Selina Song <[email protected]> * Add reverse documentation Signed-off-by: Selina Song <[email protected]> * Modify reverse test and documentation Signed-off-by: Selina Song <[email protected]> * Fix limit pushdown bug when reverse comes before head * Revert "Fix limit pushdown bug when reverse comes before head" This reverts commit 087c936. Signed-off-by: Selina Song <[email protected]> * Fix grammar, naming, and test cases. Pushdown reverted will be in 2nd PR. Signed-off-by: Selina Song <[email protected]> * Fix reverse tests: update logical plans, format with Spotless - Updated expected logical plans and Spark SQL in reverse tests - Applied Spotless to fix formatting Signed-off-by: Selina Song <[email protected]> * Fix OS version in build 3.1.0 Signed-off-by: Selina Song <[email protected]> * Add note on limitation to rst Signed-off-by: Selina Song <[email protected]> * Move explain IT to correct file, add Anonymizer test Signed-off-by: Selina Song <[email protected]> * Add reverse to index.rst Signed-off-by: Selina Song <[email protected]> --------- Signed-off-by: Selina Song <[email protected]> Co-authored-by: Selina Song <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Pass JOIN_TIME_OUT value to keepalive (#3826) * Fix JOIN_TIME_OUT Signed-off-by: Kai Huang <[email protected]> * fix style Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * fix log-rethrow Signed-off-by: Kai Huang <[email protected]> * Restructure Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fix format Signed-off-by: Kai Huang <[email protected]> * remove unused PIT Signed-off-by: Kai Huang <[email protected]> * remove reflection, add hintConfig Signed-off-by: Kai Huang <[email protected]> * Added Unit test, restructured to use existing methods Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * add IT tests Signed-off-by: Kai Huang <[email protected]> * formatting Signed-off-by: Kai Huang <[email protected]> * Trim IT Signed-off-by: Kai Huang <[email protected]> * format fix Signed-off-by: Kai Huang <[email protected]> * trim IT Signed-off-by: Kai Huang <[email protected]> * formatting Signed-off-by: Kai Huang <[email protected]> * Update legacy/src/main/java/org/opensearch/sql/legacy/query/planner/physical/node/pointInTime/PointInTime.java Co-authored-by: Louis Chu <[email protected]> Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * update .gitignore Signed-off-by: Kai Huang <[email protected]> * deletion Signed-off-by: Kai Huang <[email protected]> * restore main PIT Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Kai Huang <[email protected]> Co-authored-by: Louis Chu <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Changed seenFields to Hashset instead of LinkedHashSet Signed-off-by: Aaron Alvarez <[email protected]> * Creating a rule only for the fields/table commands to avoid any interference Signed-off-by: Aaron Alvarez <[email protected]> * This commit is in response of PR comments left by Tomo and Chen Signed-off-by: Aaron Alvarez <[email protected]> * Fixing Integration test failure Signed-off-by: Aaron Alvarez <[email protected]> * Adding anonymizer tests, wildcard unit tests, etc Signed-off-by: Aaron Alvarez <[email protected]> * Disabling Calcite for enhance fields features Signed-off-by: Aaron Alvarez <[email protected]> * Disabling automatic de-deduplication when Calcite is disabled Signed-off-by: Aaron Alvarez <[email protected]> * Adding cross-cluster IT test Signed-off-by: Aaron Alvarez <[email protected]> * Adding a dedicated Cross-cluster IT test file for Calcite Signed-off-by: Aaron Alvarez <[email protected]> * Fixing formatting issues Signed-off-by: Aaron Alvarez <[email protected]> * Improving widlcard logic and exception message Signed-off-by: Aaron Alvarez <[email protected]> * Addressing comments left by Tomo regarding wildcard logic implementation Signed-off-by: Aaron Alvarez <[email protected]> * Empty commit Signed-off-by: Aaron Alvarez <[email protected]> * Add missing udfs in v3 (#3957) * add math udfs Signed-off-by: Xinyu Hao <[email protected]> * fix decimal bug Signed-off-by: Xinyu Hao <[email protected]> * make general udf adapter Signed-off-by: Xinyu Hao <[email protected]> * add math IT Signed-off-by: Xinyu Hao <[email protected]> * fix Signed-off-by: Xinyu Hao <[email protected]> * add rst Signed-off-by: Xinyu Hao <[email protected]> * fix error Signed-off-by: Xinyu Hao <[email protected]> * change signum IT Signed-off-by: Xinyu Hao <[email protected]> * add javadoc Signed-off-by: Xinyu Hao <[email protected]> --------- Signed-off-by: Xinyu Hao <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * fix snapshot uploading (#4006) * fix snapshot uploading Signed-off-by: Kai Huang <[email protected]> * Add comment Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fix DOUBLE to STRING cast rendering zero values in scientific notation (#3982) * Fix casting double 0.0 to string Signed-off-by: Yuanchun Shen <[email protected]> * Fix float to string casting precision lost with custom FormatNumberFunction This commit fixes float to string casting by replacing the use of SqlLibraryOperators.FORMAT_NUMBER with a custom FormatNumberFunction implementation. The new implementation converts the number to a BigDecimal before formatting to preserve precision and avoid issues like 6.2 becoming 6.199999809265137. Signed-off-by: Yuanchun Shen <[email protected]> * Simplify the implementation of fp number to string cast Signed-off-by: Yuanchun Shen <[email protected]> * Update implementation of NumberToStringFunction Signed-off-by: Yuanchun Shen <[email protected]> * Cast decimal with NUMBER_TO_STRING function Signed-off-by: Yuanchun Shen <[email protected]> * Test cast decimal Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Eliminate reliance on assert in Calcite for integration test (#4016) * Move num-of-column check of in subquery ahead from RexSubQuery.java#L78 because assert is disabled in production Signed-off-by: Yuanchun Shen <[email protected]> * Increase script.context.filter.max_compilations_rate for SQLCorrectnessIT Signed-off-by: Yuanchun Shen <[email protected]> * Check script.disable_max_compilations_rate before setting context-specific compilations rate Signed-off-by: Yuanchun Shen <[email protected]> * Refactor: remove some methods in tests to upper level to reduce duplication Signed-off-by: Yuanchun Shen <[email protected]> * Replace plugin-level setting strings with private test-specific ones Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Prevent aggregation push down when it has inner filter (#4002) * Prevent aggregation push down when it has inner filter Signed-off-by: Heng Qian <[email protected]> * Fix IT & Remove log Signed-off-by: Heng Qian <[email protected]> * Fix 4009 Signed-off-by: Heng Qian <[email protected]> * Fix IT Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fix span on negative timestamp (#4017) * Fix span on negative timestamp Signed-off-by: Heng Qian <[email protected]> * Fix span on negative timestamp Signed-off-by: Heng Qian <[email protected]> * typo Signed-off-by: Heng Qian <[email protected]> * Refine code Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Skip script encoding when run explain with 'extended' (#3930) * No need to decode script when run explain command Signed-off-by: Lantao Jin <[email protected]> * address comment Signed-off-by: Lantao Jin <[email protected]> * Do not encoding when explain format is 'extended' Signed-off-by: Lantao Jin <[email protected]> * Rename the thread local var Signed-off-by: Lantao Jin <[email protected]> * Fix IT after merge main Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Implement type checking for aggregation functions with Calcite (#4024) * Remove getTypeChecker from FunctionImp interface Signed-off-by: Yuanchun Shen <[email protected]> * Refactor registerExternalFunction to registerExternalOperator Signed-off-by: Yuanchun Shen <[email protected]> * Do not register GEOIP function if got incompatible client Signed-off-by: Yuanchun Shen <[email protected]> * Create scaffold for type checking of aggregation functions Signed-off-by: Yuanchun Shen <[email protected]> * Add type checkers for aggregation functions Signed-off-by: Yuanchun Shen <[email protected]> * Test type checking for aggregation functions Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Allow equal expression as a function argument (#4001) * Remove named function arg from functions other than table functions Signed-off-by: Yuanchun Shen <[email protected]> * Test eval if function with equal as condition Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Push down IP comparison as range query with Calcite (#3959) * Add reverse op for compare ip to support pushdown Signed-off-by: Yuanchun Shen <[email protected]> * Pushdown ip comparison Signed-off-by: Yuanchun Shen <[email protected]> * Refactor CompareIpFunction to use SqlKind directly Signed-off-by: Yuanchun Shen <[email protected]> * Simplify the overriding of reverse() for IP comparators Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * eval sum, avg implementation (#3986) Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fix PPL eval command string concatenation with + operator (#4020) * eval command support Signed-off-by: Kai Huang <[email protected]> * improvment Signed-off-by: Kai Huang <[email protected]> * Refactor Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fix CI Signed-off-by: Kai Huang <[email protected]> * fixes Signed-off-by: Kai Huang <[email protected]> * fix Signed-off-by: Kai Huang <[email protected]> * Add IT Signed-off-by: Kai Huang <[email protected]> * remove redundant tests Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Support script push down on text field (#4010) * Support script push down on text field Signed-off-by: Heng Qian <[email protected]> * Fix IT Signed-off-by: Heng Qian <[email protected]> * Add UT for struct type push down Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Enhance sort command in PPL (#3934) * enhance sort command Signed-off-by: Ritvi Bhatt <[email protected]> * update failing tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix formatting Signed-off-by: Ritvi Bhatt <[email protected]> * add integ tests Signed-off-by: Ritvi Bhatt <[email protected]> * update documentation Signed-off-by: Ritvi Bhatt <[email protected]> * fix failing test Signed-off-by: Ritvi Bhatt <[email protected]> * update default and tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix tests Signed-off-by: Ritvi Bhatt <[email protected]> * update analyzer test Signed-off-by: Ritvi Bhatt <[email protected]> * update reverse sort direction Signed-off-by: Ritvi Bhatt <[email protected]> * update formatting Signed-off-by: Ritvi Bhatt <[email protected]> * update docs Signed-off-by: Ritvi Bhatt <[email protected]> * add javadoc Signed-off-by: Ritvi Bhatt <[email protected]> * add tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix failing tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix failing tests Signed-off-by: Ritvi Bhatt <[email protected]> * update integ tests for query size limit change Signed-off-by: Ritvi Bhatt <[email protected]> * add explainit for desc and type cast Signed-off-by: Ritvi Bhatt <[email protected]> * add tests for desc Signed-off-by: Ritvi Bhatt <[email protected]> * fix formatting Signed-off-by: Ritvi Bhatt <[email protected]> * make count optional Signed-off-by: Ritvi Bhatt <[email protected]> * add cross cluster tests Signed-off-by: Ritvi Bhatt <[email protected]> * fix tests Signed-off-by: Ritvi Bhatt <[email protected]> * normalize count in AST node Signed-off-by: Ritvi Bhatt <[email protected]> * default null count to 0 Signed-off-by: Ritvi Bhatt <[email protected]> * update logicalsort default constructor Signed-off-by: Ritvi Bhatt <[email protected]> --------- Signed-off-by: Ritvi Bhatt <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Add example for String concat in eval.rst (#4075) * Add example for String concat in eval.rst Signed-off-by: Kai Huang <[email protected]> * mention calcite enabling Signed-off-by: Kai Huang <[email protected]> --------- Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Support pushdown dedup with Calcite (#3972) * Support pushdown dedup with Calcite Signed-off-by: Lantao Jin <[email protected]> * Fix IT Signed-off-by: Lantao Jin <[email protected]> * Address comments Signed-off-by: Lantao Jin <[email protected]> * Fix flaky test Signed-off-by: Lantao Jin <[email protected]> * Address comment Signed-off-by: Lantao Jin <[email protected]> * delete useless codes Signed-off-by: Lantao Jin <[email protected]> * Add more ITs Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Fix CI failure because of plan having changed (#4077) Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> * Empty commit Signed-off-by: Aaron Alvarez <[email protected]> * Empty commit Signed-off-by: Aaron Alvarez <[email protected]> --------- Signed-off-by: Aaron Alvarez <[email protected]> Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Songkan Tang <[email protected]> Signed-off-by: Simeon Widdis <[email protected]> Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Peng Huo <[email protected]> Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Anas Alkouz <[email protected]> Signed-off-by: opensearch-ci <[email protected]> Signed-off-by: opensearch-ci-bot <[email protected]> Signed-off-by: Selina Song <[email protected]> Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Xinyu Hao <[email protected]> Signed-off-by: Ritvi Bhatt <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> Co-authored-by: Aaron Alvarez <[email protected]> Co-authored-by: Lantao Jin <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]> Co-authored-by: Songkan Tang <[email protected]> Co-authored-by: Simeon Widdis <[email protected]> Co-authored-by: Yuanchun Shen <[email protected]> Co-authored-by: Peng Huo <[email protected]> Co-authored-by: qianheng <[email protected]> Co-authored-by: Anas Alkouz <[email protected]> Co-authored-by: Zelin Hao <[email protected]> Co-authored-by: Sayali Gaikawad <[email protected]> Co-authored-by: opensearch-ci <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Selina Song <[email protected]> Co-authored-by: Selina Song <[email protected]> Co-authored-by: Kai Huang <[email protected]> Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Xinyu Hao <[email protected]> Co-authored-by: ritvibhatt <[email protected]> (cherry picked from commit f16f1c0) * Fixing compatibility issues Signed-off-by: Aaron Alvarez <[email protected]> * Fixing JDK compatibility issues Signed-off-by: Aaron Alvarez <[email protected]> * fixing java 11 compatibility issues Signed-off-by: Aaron Alvarez <[email protected]> --------- Signed-off-by: Aaron Alvarez <[email protected]> Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Vamsi Manohar <[email protected]> Signed-off-by: Songkan Tang <[email protected]> Signed-off-by: Simeon Widdis <[email protected]> Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: Peng Huo <[email protected]> Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Anas Alkouz <[email protected]> Signed-off-by: opensearch-ci <[email protected]> Signed-off-by: opensearch-ci-bot <[email protected]> Signed-off-by: Selina Song <[email protected]> Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Kai Huang <[email protected]> Signed-off-by: Xinyu Hao <[email protected]> Signed-off-by: Ritvi Bhatt <[email protected]> Signed-off-by: Aaron Alvarez <[email protected]> Co-authored-by: Aaron Alvarez <[email protected]> Co-authored-by: Lantao Jin <[email protected]> Co-authored-by: Vamsi Manohar <[email protected]> Co-authored-by: Songkan Tang <[email protected]> Co-authored-by: Simeon Widdis <[email protected]> Co-authored-by: Yuanchun Shen <[email protected]> Co-authored-by: Peng Huo <[email protected]> Co-authored-by: qianheng <[email protected]> Co-authored-by: Anas Alkouz <[email protected]> Co-authored-by: Zelin Hao <[email protected]> Co-authored-by: Sayali Gaikawad <[email protected]> Co-authored-by: opensearch-ci <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Selina Song <[email protected]> Co-authored-by: Selina Song <[email protected]> Co-authored-by: Kai Huang <[email protected]> Co-authored-by: Louis Chu <[email protected]> Co-authored-by: Xinyu Hao <[email protected]> Co-authored-by: ritvibhatt <[email protected]>

Append limit operator for QUEERY_SIZE_LIMIT

e8d60f8

Signed-off-by: Heng Qian <[email protected]>

qianheng-aws requested review from GumpacG, LantaoJin, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, seankao-az and ykmr1224 as code owners July 30, 2025 08:32

qianheng-aws added enhancement New feature or request PPL Piped processing language calcite calcite migration releated labels Jul 30, 2025

LantaoJin reviewed Jul 30, 2025

View reviewed changes

qianheng-aws added 4 commits July 31, 2025 16:33

Add LogicalSystemLimit

f272238

Signed-off-by: Heng Qian <[email protected]>

Revert part of opensearch-project#3880

7217650

Signed-off-by: Heng Qian <[email protected]>

Merge remote-tracking branch 'refs/remotes/origin/main' into QuerySiz…

ebb539d

…ePushDown

Fix IT after merging main

7417f12

Signed-off-by: Heng Qian <[email protected]>

LantaoJin approved these changes Jul 31, 2025

View reviewed changes

penghuo reviewed Jul 31, 2025

View reviewed changes

penghuo approved these changes Jul 31, 2025

View reviewed changes

penghuo merged commit 5316c0a into opensearch-project:main Jul 31, 2025
23 checks passed

qianheng-aws added the backport 2.19-dev label Aug 1, 2025

opensearch-trigger-bot bot added the backport-failed label Aug 1, 2025

LantaoJin added the backport-manually Filed a PR to backport manually. label Aug 6, 2025

opensearch-project deleted a comment from opensearch-trigger-bot bot Aug 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Append limit operator for QUEERY_SIZE_LIMIT #3940

Append limit operator for QUEERY_SIZE_LIMIT #3940

Uh oh!

qianheng-aws commented Jul 30, 2025

Uh oh!

LantaoJin Jul 30, 2025

Uh oh!

qianheng-aws Jul 30, 2025 •

edited

Loading

Uh oh!

penghuo Jul 30, 2025

Uh oh!

LantaoJin Jul 31, 2025

Uh oh!

qianheng-aws Jul 31, 2025 •

edited

Loading

Uh oh!

penghuo Jul 31, 2025

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Aug 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Append limit operator for QUEERY_SIZE_LIMIT #3940

Append limit operator for QUEERY_SIZE_LIMIT #3940

Uh oh!

Conversation

qianheng-aws commented Jul 30, 2025

Description

Related Issues

Check List

Uh oh!

LantaoJin Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

qianheng-aws Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

penghuo Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

LantaoJin Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

qianheng-aws Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

penghuo Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Aug 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

qianheng-aws Jul 30, 2025 •

edited

Loading

qianheng-aws Jul 31, 2025 •

edited

Loading