Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions #4599

penghuo · 2025-10-18T05:09:09Z

Description

Expand the Calcite coercion logic, so functions and aggregation operators automatically cast compatible strings to numeric or boolean overloads, and return the coerced Rex nodes to the execution pipeline.
Test cover in 4356.yaml.
Document implicit conversion rules for PPL
To revieiwer
- Add TypeChecker for SUBTRACT, CEIL/CEILING, FLOOR, and ROUND. It is a workaround to support sequence CompositeOperandTypeChecker.
- To migrate user issue, current CoercionUtils change is hack, refactor the code in following PRs to include leverage TypeCoercionImpl

Related Issues

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Peng Huo <[email protected]>

core/src/main/java/org/opensearch/sql/expression/function/CoercionUtils.java

Signed-off-by: Peng Huo <[email protected]>

core/src/main/java/org/opensearch/sql/expression/function/CoercionUtils.java

core/src/main/java/org/opensearch/sql/calcite/CalciteRexNodeVisitor.java

Signed-off-by: Peng Huo <[email protected]>

core/src/main/java/org/opensearch/sql/expression/function/PPLFuncImpTable.java

Signed-off-by: Peng Huo <[email protected]>

core/src/main/java/org/opensearch/sql/expression/function/CoercionUtils.java

core/src/main/java/org/opensearch/sql/expression/function/PPLFuncImpTable.java

opensearch-trigger-bot · 2025-10-23T15:41:58Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4599-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 6f028a3ef18b92ecd1769e788fe867bdd74e3544
# Push it to GitHub
git push --set-upstream origin backport/backport-4599-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4599-to-2.19-dev.

…ions (opensearch-project#4599) Signed-off-by: Peng Huo <[email protected]>

…ARSE Command Extraction (#4599) (#4650) * Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions (#4599) --------- Signed-off-by: Peng Huo <[email protected]>

* default-main: (34 commits) Enhance dynamic source clause to support only metadata filters (opensearch-project#4554) Make nested alias type support referring to outer context (opensearch-project#4673) Update big5 ppl queries and check plans (opensearch-project#4668) Support push down sort after limit (opensearch-project#4657) Use table scan rowType in filter pushdown could fix rename issue (opensearch-project#4670) Fix: Support Alias Fields in MIN, MAX, FIRST, LAST, and TAKE Aggregations (opensearch-project#4621) Fix bin nested fields issue (opensearch-project#4606) Add `per_minute`, `per_hour`, `per_day` function support (opensearch-project#4531) Pushdown sort aggregate metrics (opensearch-project#4603) Followup: Change ComparableLinkedHashMap to compare Key than Value (opensearch-project#4648) Mitigate the CI failure caused by 500 Internal Server Error (opensearch-project#4646) Allow renaming group-by fields to existing field names (opensearch-project#4586) Publish internal modules separately for downstream reuse (opensearch-project#4484) Revert "Update grammar files and developer guide (opensearch-project#4301)" (opensearch-project#4643) Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions (opensearch-project#4599) Replace all dots in fields of table scan's PhysType (opensearch-project#4633) Return comparable LinkedHashMap in `valueForCalcite()` of ExprTupleValue (opensearch-project#4629) Refactor JsonExtractAllFunctionIT and MapConcatFunctionIT (opensearch-project#4623) Pushdown case function in aggregations as range queries (opensearch-project#4400) Update GEOIP function to support IP types as input (opensearch-project#4613) ... # Conflicts: # docs/user/ppl/functions/conversion.rst

* default-main: (34 commits) Enhance dynamic source clause to support only metadata filters (opensearch-project#4554) Make nested alias type support referring to outer context (opensearch-project#4673) Update big5 ppl queries and check plans (opensearch-project#4668) Support push down sort after limit (opensearch-project#4657) Use table scan rowType in filter pushdown could fix rename issue (opensearch-project#4670) Fix: Support Alias Fields in MIN, MAX, FIRST, LAST, and TAKE Aggregations (opensearch-project#4621) Fix bin nested fields issue (opensearch-project#4606) Add `per_minute`, `per_hour`, `per_day` function support (opensearch-project#4531) Pushdown sort aggregate metrics (opensearch-project#4603) Followup: Change ComparableLinkedHashMap to compare Key than Value (opensearch-project#4648) Mitigate the CI failure caused by 500 Internal Server Error (opensearch-project#4646) Allow renaming group-by fields to existing field names (opensearch-project#4586) Publish internal modules separately for downstream reuse (opensearch-project#4484) Revert "Update grammar files and developer guide (opensearch-project#4301)" (opensearch-project#4643) Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions (opensearch-project#4599) Replace all dots in fields of table scan's PhysType (opensearch-project#4633) Return comparable LinkedHashMap in `valueForCalcite()` of ExprTupleValue (opensearch-project#4629) Refactor JsonExtractAllFunctionIT and MapConcatFunctionIT (opensearch-project#4623) Pushdown case function in aggregations as range queries (opensearch-project#4400) Update GEOIP function to support IP types as input (opensearch-project#4613) ... Signed-off-by: Asif Bashar <[email protected]>

…ions (opensearch-project#4599) Signed-off-by: Peng Huo <[email protected]>

penghuo added PPL Piped processing language enhancement New feature or request labels Oct 18, 2025

penghuo force-pushed the issues/4356 branch 2 times, most recently from 1d3ebf1 to 3902dcd Compare October 20, 2025 18:27

Init version

28c4777

Signed-off-by: Peng Huo <[email protected]>

penghuo force-pushed the issues/4356 branch from 3902dcd to 28c4777 Compare October 20, 2025 22:15

penghuo added 3 commits October 20, 2025 15:17

Update doctest

202ac3e

Signed-off-by: Peng Huo <[email protected]>

Merge branch 'main' into issues/4356

4901caf

Update Doc

cecd2db

Signed-off-by: Peng Huo <[email protected]>

yuancu reviewed Oct 21, 2025

View reviewed changes

Support MINUS, CEIL, FLOOR, ROUND

1066422

Signed-off-by: Peng Huo <[email protected]>

penghuo self-assigned this Oct 22, 2025

penghuo marked this pull request as ready for review October 22, 2025 02:48

penghuo requested review from GumpacG, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, ps48, seankao-az, vamsimanohar and ykmr1224 as code owners October 22, 2025 02:48

penghuo requested review from LantaoJin, RyanL1997, noCharger and qianheng-aws as code owners October 22, 2025 02:48

penghuo commented Oct 22, 2025

View reviewed changes

core/src/main/java/org/opensearch/sql/expression/function/CoercionUtils.java Show resolved Hide resolved

qianheng-aws reviewed Oct 22, 2025

View reviewed changes

core/src/main/java/org/opensearch/sql/expression/function/CoercionUtils.java Show resolved Hide resolved

core/src/main/java/org/opensearch/sql/expression/function/CoercionUtils.java Outdated Show resolved Hide resolved

qianheng-aws reviewed Oct 22, 2025

View reviewed changes

core/src/main/java/org/opensearch/sql/calcite/CalciteRexNodeVisitor.java Outdated Show resolved Hide resolved

penghuo added 3 commits October 22, 2025 08:49

Fix window function bug

cde2238

Signed-off-by: Peng Huo <[email protected]>

Fix IT

97235a3

Signed-off-by: Peng Huo <[email protected]>

Update

f40ba6f

Signed-off-by: Peng Huo <[email protected]>

penghuo commented Oct 22, 2025

View reviewed changes

core/src/main/java/org/opensearch/sql/expression/function/PPLFuncImpTable.java Show resolved Hide resolved

Revert log level change

3e7788c

Signed-off-by: Peng Huo <[email protected]>

yuancu approved these changes Oct 23, 2025

View reviewed changes

core/src/main/java/org/opensearch/sql/expression/function/CoercionUtils.java Show resolved Hide resolved

core/src/main/java/org/opensearch/sql/expression/function/PPLFuncImpTable.java Show resolved Hide resolved

qianheng-aws approved these changes Oct 23, 2025

View reviewed changes

penghuo added the backport 2.19-dev label Oct 23, 2025

penghuo merged commit 6f028a3 into opensearch-project:main Oct 23, 2025
35 checks passed

opensearch-trigger-bot bot added the backport-failed label Oct 23, 2025

penghuo added a commit to penghuo/os-sql that referenced this pull request Oct 23, 2025

Support Automatic Type Conversion for REX/SPATH/PARSE Command Extract…

870ef6e

…ions (opensearch-project#4599) Signed-off-by: Peng Huo <[email protected]>

penghuo mentioned this pull request Oct 23, 2025

[Backport 2.19-dev] Support Automatic Type Conversion for REX/SPATH/PARSE Command Extraction (#4599) #4650

Merged

LantaoJin added the backport-manually Filed a PR to backport manually. label Oct 24, 2025

expani pushed a commit to vinaykpud/sql that referenced this pull request Nov 4, 2025

Support Automatic Type Conversion for REX/SPATH/PARSE Command Extract…

b7b66f8

…ions (opensearch-project#4599) Signed-off-by: Peng Huo <[email protected]>

yuancu mentioned this pull request Nov 6, 2025

Support chart command in PPL #4579

Merged

22 tasks

dai-chen mentioned this pull request Nov 11, 2025

[BUG] SAFE_CAST function is not translated to Spark-compatible cast #4778

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions #4599

Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions #4599

Uh oh!

penghuo commented Oct 18, 2025 •

edited by yuancu

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Oct 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions #4599

Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions #4599

Uh oh!

Conversation

penghuo commented Oct 18, 2025 • edited by yuancu Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Oct 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

penghuo commented Oct 18, 2025 •

edited by yuancu

Loading